Machine learning with requirements: A manifesto

1. Introduction

In recent years, machine learning has made great advancements that have been at the root of many breakthroughs in different application domains. For example, AlphaFold [63] is a deep learning model that solved the “protein folding problem”, a grand challenge in the field of biology for more than half a century, while Halicin [71] is the first antibiotic discovered using machine learning, which could help in the battle against bacterial resistance. Results like those above, though ground-breaking, overshadow the dangers that come with the careless use of machine learning models in critical applications. Indeed, even though such models report an astonishingly high performance in terms of accuracy (or alternatively chosen metric), they do not give any guarantee that the model will not have any unintended behaviour when used in practice. Indeed, as pointed out by Rudin in [59], there have been several notable instances of machine learning failures. For example, individuals have been wrongly denied parole [81], deep learning models have inaccurately reported that highly polluted air was safe to breathe [47], and there has been generally inefficient use of limited resources across fields such as medicine, criminal justice, and finance [76]. Such problems in the behavior of the model are often rooted in the quality of the dataset used for training the model. This is, for example, what happened in the early stages of the Covid-19 pandemic, where (as reported in [58]) a common chest scan dataset from [35] made of pediatric scans, was used as control group against Covid-19 positive scans: as a result, instead of Covid-19 detectors, adult/child classifiers were built. Unintended outcomes like the ones described above (i) are particularly dangerous in safety-critical applications, where even a single unforeseen mistake can lead to dramatic consequences, and (ii) undermine human confidence in the models themselves, thus slowing down their adoption.

In this paper, we argue that requirements specification and verification can go a long way to make machine learning models even more fitting to the real world, by reducing the risk of getting potentially dangerous unintended behaviors. We start from the simple observation that unintended behaviors correspond to the model violating some requirements which may be known even before the data collection and model development start. To support this claim, we consider two examples, one in healthcare, and the other one in autonomous driving, in which (i) some requirements are known in advance, (ii) machine learning models have very high performance in terms of accuracy (or other selected metric), and (iii) despite the positive performance, the outcome of standardly developed machine learning models often violates the known requirements with possible dramatic consequences given the criticality of the application domains. Although we consider just two examples, we believe that analogous considerations apply to most application domains given (i) the body of knowledge developed over the years in many application domains, which can be translated in corresponding requirements (see, e.g., [22,33,50,70]), (ii) the impressive results in performance obtained by machine learning models, which will continue to push for their adoption despite the possible unintended behaviors (see, e.g., [5,24]), and (iii) the fact that it is not surprising that the resulting model will violate the requirements if they are not taken into account during data collection and model development. Refining the last observation, it is not surprising that a machine learning model has an unintended behavior if, before its deployment, the data and the model itself are not somehow verified against the requirement capturing the intended behavior. We therefore claim that the requirements’ definition should precede and involve the entire machine learning model development cycle, including the dataset construction. In particular, we propose a novel pyramid machine learning development process, which (i) highlights how the requirement definition can help improve every single step of the standard machine learning model development process, and (ii) takes into account that some requirement adjustment might be necessary because of difficulties and/or discoveries that emerge during the development process of machine learning models, especially given their data-driven nature. In line with what has been argued in the Machine Learning Operations (MLOps) field [7,25,32,36,74], the above proposal can be seen as a call to adapt and adopt the general methodologies used in software engineering for generic system development, where it is well known (i) that the later the requirement are taken into account the higher the cost is to repair the system, and also (ii) that the requirement definition is an iterative process in which requirements affect the system development and vice versa, given that during the system development some requirement may be newly discovered and/or adjusted if not canceled (see, e.g., [52,67,68]). We will ground the above facts in the specific machine learning model development pipeline, highlighting the benefits of adopting requirements in the different phases.

Our proposal represents a significant shift from the traditional “performance-driven” machine learning development pipeline, which solely concentrates on how to get better performance, measured in terms of accuracy or alternatively chosen metric. Indeed, we are proposing a “requirements-driven” machine learning development pipeline in which performance is just one of the many requirements that the model has to satisfy. This obviously might make the development process more complex, because different requirements might be contradictory with one another: a problem that is well known in software engineering [68]. For example, at a high level of abstraction, complex models will likely have a better performance at the possible price of being less explainable and/or verifiable and/or sustainable. Analogously, it might be the case that satisfying certain fairness properties (e.g., equalized odds or demographic parity) is preferable to a higher performance. Independently from the properties that we wish for the model, it is clear that (i) the sooner the requirements are made explicit the better, and (ii) taking into account the requirements in the dataset definition and model construction will likely lead to models that will be easier to verify with respect to the stated requirements. This proposal thus aligns with the efforts in many other machine learning fields (see, e.g., fairness, robustness, explainability, sustainability, and safety), where it is argued that performance should not be the only factor to evaluate a model. However, in the aforementioned fields, there is often either no formal or unique definition of what is the desired property (see, e.g., interpretable) and, even when there is, its incorporation in the machine learning pipeline requires a customised loss/architecture that requires a lot of time and knowledge to design. Here, on the other hand, we will take a step further and we will claim that the requirements will ideally need to be formally expressed. This will be pivotal to their wide applicability in the machine learning domain. If the requirements were to be formally defined, then we could not only assign a unique meaning to each requirement, but also create automatic procedures to incorporate such requirements into the machine learning development pipeline. To this end, the field of Neuro-symbolic AI is well-positioned to lead this research. Researchers in this field possess the expertise to (i) encode background knowledge and desiderata formally, (ii) understand the inner workings of machine learning models, and (iii) automatically integrate such requirements into the topology and loss functions of various models. For example, in recent years researchers in the Neuro-symbolic AI field have developed models that are guaranteed-by-design to be compliant with a set of given requirements expressed as logical constraints (see, e.g., [3,21,22,28]): in these works, it has been shown how the requirements can be incorporated in the model, which automatically satisfies them, and have a positive impact also on performance.

Given the above, our proposal enjoys the same spirit of works such as [19,49,61], which advocate for a more structured approach to machine learning model development. In particular, Gebru et al. in [19] propose documentation guidelines for new datasets, Mitchell et al. in [49] advocate for standardized reporting of models, including training data, performance measures, and limitations, while Seedat et al. [61] propose an actionable checklist-style framework to elicit data-centric considerations at different stages of the development pipeline.

The remainder of the paper is structured as follows. First, in Section 2, we consider examples from the healthcare and autonomous driving application domains in order to show that in these domains (i) requirements arise naturally, (ii) machine learning models are or can be fruitfully deployed, and (iii) neglecting the requirements can have dramatic consequences. Secondly, in Section 3, we show how the requirements definition can be fruitfully integrated into the standard machine learning development pipeline, impacting all the phases in the pipeline. Finally, we have the conclusions and possible plans for the road ahead in Section 5.

2. The need for requirements

The standard machine learning pipeline naturally tends to have a sequential nature. The main steps that compose it are:

Data curation, in which the dataset for training the model is created, encompassing the following phases: (i) data collection, (ii) data pre-processing, (iii) data augmentation, and (iv) data quality evaluation;

OPEN IN VIEWER

Figure 1.

Visualization of the standard machine learning pipeline.

We now focus on two application domains, healthcare and autonomous driving, and we show how requirements arise naturally in both applications domains, how applying machine learning techniques has already brought and can bring tremendous advantages to the fields, and how deploying machine learning models in these fields without explicitly taking into account the requirements can lead to unexpected behaviors corresponding to violations of the requirements. Though we consider just two application domains, we believe that analogous considerations and results hold virtually in any application domain, given (i) the body of knowledge developed over the years in any application domain, which can be translated in corresponding requirements, (ii) the impressive results in performance obtained by machine learning models which will continue to push for their adoption despite the possible unintended behaviors, and (iii) the impossibility to certify/rule out the absence of undesired behaviors without explicitly spelling out the corresponding requirements and the testing/verification of the model against the requirements themselves.

2.1. Autonomous driving

In recent years, the developments in machine learning, and in particular in computer vision, have fuelled the hopes of autonomous vehicles being finally in reach. However, every so often, such dreams are shattered by car crashes that, in some cases, have injured or even killed people. For example, in March 2018, an autonomous vehicle developed and tested on public roads by Uber’s Advanced Technologies Group fatally injured a pedestrian who was pushing their bicycle across the street outside of a designated crossing area [51]. The most striking characteristic of this accident is the fact that the car did not make any attempt to break and/or to avoid the pedestrian [40]. Unfortunately, this is not an isolated incident, as nearly 400 car crashes involving autonomous vehicles have been reported in the United States over a period of only ten months in 2022 [9]. As reported in [9], these vehicles in order to take their decisions rely, among others, on computer vision models, which if trained to simply maximise their performance (following the standard machine learning development pipeline) might fail to abide to even the simplest requirements.

OPEN IN VIEWER

Figure 2.

(a) and (b) show the predictions made by the I3D model (with θ = 0.5 ) for the same traffic light and just one frame apart. (c) shows the number of predictions that violate at least one requirement when varying θ; (c) is from [22].

To exemplify the problem described above, we consider the ROAD-R dataset [22], which consists of (i) 22 relatively long (∼8 minutes each) videos annotated with road events, i.e., a sequence of frame-wise bounding boxes linked in time, each labelled with the agent in the bounding box, together with its action(s) and location(s), and (ii) 243 requirements expressed in propositional logic stating what is an admissible road event.1 The possible labels associated to each bounding box are 41, and the requirements state which combinations of labels a model can output. Hence, for instance, a requirement states that a traffic light cannot be red and green at the same, while another states that a traffic light cannot move. Given ROAD-R, six state-of-the-art temporal feature learning architectures (I3D [10], C2D [79], RCGRU [29], RCLSTM [29], RCN [66] and SlowFast [16]) as part of a 3D-RetinaNet model [65] (with a 2D-ConvNet backbone made of Resnet50 [26]) for event detection, have been trained. These models take as input a set of consecutive frames, and for each frame they output (i) a set of bounding boxes, and (ii) a set of labels for each bounding box. Such labels are decided in the standard way: for each of the 41 labels the model outputs a number o ∈ [ 0 , 1 ] , and if o > θ then the label is returned, otherwise it is not. θ is a user-defined threshold. An example of prediction is given in Figure 2a, where the prediction for the depicted bounding box is { Traffic Light , Red Traffic Light } . The surprising finding of the work [22] is that, as shown in Figure 2c, no matter the chosen threshold θ, at least 89% of the predictions violate at least one requirement. Even more, some of the predictions may violate requirements corresponding to common knowledge possessed by humans, making the prediction difficult to interpret and manage. For example, if we consider the prediction in Figure 2b, done by the same model that made the prediction in Figure 2a, for the same traffic light just one frame later, then we see that not only the prediction is wrong, but it also violates the common knowledge (in this case corresponding also to a formally stated requirement) that the traffic light cannot be red and green at the same time. Such prediction, if not further elaborated by the system controlling the vehicle could indeed have dramatic consequences. For this reason, we expect that any system controlling the vehicle will have mechanisms in place to handle the predictions that violate the requirements. However, if the requirements are incorporated into and verified by the entire system (as must be the case in this setting), it is far from clear why they are standardly neglected in the process of building the machine learning component of the system. Indeed, none of the six evaluated systems is able to handle requirements on their inputs and/or outputs. Even more, as reported in [22], the dataset used for training the model violates some of the requirements (like the fact that it is not possible for a vehicle to be both incoming and outgoing), which surely will have to be incorporated into any real application of the models.2

OPEN IN VIEWER

Figure 3.

Figure 3a shows two examples of the predictions made by different classification models for the same patient at different time horizons. Figure 3b shows the proportion of samples for which the predictions violate the requirement for risk to be increasing for longer time horizons.

3. The pyramid model

In the previous section, we have shown the standard performance-driven machine learning development pipeline, in which the requirements are traditionally neglected, and, with the means of two examples, we have highlighted how such negligence can have dire consequences. This automatically calls for the inclusion of the requirements in the development process, which are at the core of any development pipeline proposed in the field of software engineering. We thus propose to adapt the general methodologies used in software engineering for generic system development, where it is well known that the later the requirements are taken into account the higher the cost is to repair the system. In order to adapt such methodologies, we need to take into account that, differently from standard software, machine learning models are learnt from data, and thus we have less control over the behaviour of the model, which entails not only that at design time it is impossible to predict the future behaviour of the model, but also, at testing time, if a bug is found, then the steps necessary to fix it are entirely different. A development pipeline for a machine learning model thus needs to take into account that:

data are central to the process, as the quality of the model heavily depends on the quality of the collected data,

OPEN IN VIEWER

Figure 4.

Visualization of the pyramid model. The full arrows stand for the standard procedural processes, while the dotted arrows show that the requirements impact every stage of the process.

Given the above, we propose the pyramid machine learning development model in Figure 4, in which the requirement definition phase is (as expected) put at the beginning of the development process and in which the pyramid shape allows to better highlight the close relationship that must exist between the requirement definition phase and all the others in the pipeline. Ideally, the requirements should be annotated in a formal way, to allow for automatic (i) parsing, (ii) integration in every step of the pipeline, and (iii) testing and verification. Consider the grey backward lines. These represent the fact that if something unexpected happens at any point of the pipeline (e.g., the model does have some unintended behavior during the testing phase), then it is necessary to go back and first check if the defined requirements were suitable or need to be updated, then if the collected data are representative of the phenomenon that we want to capture, and so on. Obviously, the later in the pipeline we realise that there is a bug, the more costly it is, and this is captured in Figure 4 by the length of the path to cover as we advance in the pipeline. Consider now the dotted arrows, which illustrate the fact that the requirements can help in shaping each of the development steps, and in particular:

As already stated in the introduction, researchers in the Neuro-symbolic AI field find themselves in the unique position of being able to not only understand the deep impact that the violation of the requirements can have on the trust the public poses in the AI models, but also leverage the techniques developed in the traditional AI field to fill the gaps existing in the current state-of-the-art machine learning models.

As a testament of this fact, they have already been at the forefront of this line of research, with researchers developing methods which are guaranteed to be compliant with a set of requirements over the output space expressed as a logic program [42,43], in propositional logic [4,23], as linear inequalities [18,70] and even as quantifier free linear arithmetic formula over the rationals [28]. If we think about it, this was already an incredible leap forward, as up to that point specialised methods were being developed for different types of requirements. As an example, we can think of the field of hierarchical multi-label classification, where the labels are organised in a hierarchy and every prediction had to be compliant with such hierarchy. Many different models (see, e.g., [60,64,77]) were developed to deal with these very simple constraints. If the requirement were to be changed even in the slightest, then those methods would no longer be applicable. On the contrary, thanks to Neuro-symbolic AI we can now build models compliant by-design with any set of requirements which can be expressed with a pre-defined syntax (e.g., propositional logic).

We expect that in the future the expressivity of the requirements studied will increase and will thus encompass more scenarios. For example, requirements expressing the relations existing over multiple data points might be expressed in first order logic (FOL) or requirements over the behaviour of an auto-regressive model might be expressed in linear temporal logic over finite traces ( LTL f ) [20]. Then, thanks to Neuro-symbolic AI, we will not have to study each requirement singularly, but we will be able to develop novel models able to deal with entire classes of requirements altogether. Note, that a step in this direction has already been made, with Neuro-symbolic AI models able to incorporate requirements expressed in FOL [8] and to deal with auto-regressive models [2]. Despite the fact that these models cannot guarantee the satisfaction of the constraints, they denote the pro-activeness of Neuro-symbolic AI researchers to tackle more and more complex problems. Further, Neuro-symbolic AI could even be used to accelerate the phase of requirements elicitation, by using automatic methods able to discover constraints directly from the data (see, e.g., [53,56]) thus making the integration of the requirements in any pipeline even easier.

Finally, Neuro-symbolic AI has already shown very promising results in dealing with even more complex cases than the standard fully-supervised ones. For example, many works have been developed to be able to exploit (and sometimes impose) the requirements in semi-supervised settings (see, e.g., [8]), in partial label settings (see, e.g., [17]) and sometimes even in fully unsupervised settings (see, e.g., [69]). Naturally, these scenarios pose numerous challenges, including the exploitation of reasoning shortcuts by Neuro-symbolic AI models [45]; however, there is already research underway to mitigate this issue [44,78].

5. Summary and outlook

In this paper, we have shown that even though applying machine learning to a given problem can bring great benefits, applying it without specifying and taking into account the requirements of the problem can lead to unexpected behaviors and, possibly, dramatic consequences. We have thus proposed a new machine learning development pipeline in which the requirements definition phase is explicitly incorporated at the beginning of the process, causing a deep connection between the requirements definition and all the other phases in the development process, in which (as standard in software engineering) the former may affect the latter and vice versa. From this perspective, this paper can be seen as a call to adopt the general methodologies used in software engineering for generic system development to the specific field of machine learning, highlighting the risks of not doing so. We believe that in many cases the benefits of explicitly defining the requirements and taking them into account in the other phases outweigh their cost. It is also clear that the requirement definition and exploitation is unavoidable when machine learning models are used as stand-alone systems in domains with strict requirements to satisfy (e.g., in safety critical domains) or when used as part of larger systems with a given set of stringent requirements (e.g., memory usage; see, e.g., [57]). This view is supported by the recent trends in machine learning in which performance is just one of the requirements to satisfy. In particular, work in the fields of interpretability (see, e.g., [11,15],Imrie2023multiple), fairness (see, e.g., [48,82]) and robustness (see, e.g., [34,80]) all privilege certain characteristics of the model over performance, and all these works roots their motivation in the necessity to take into account requirements emerging from the respective application domain. However, how to define and fully exploit the domain knowledge expressed by requirements in the development process is still largely an open question, as many technical challenges need to be overcome. To this end, the Neuro-symbolic AI field can be seen as a bridge between standard machine learning and software engineering, as researchers in the field have already started developing techniques to create models that are built not only by the data, but that take explicitly as input also other types of knowledge, like formally stated requirements and use them either to design the model (see, e.g., [21,28]), or to incorporate the requirements in the loss function (see, e.g., [8,14]).

Despite the huge amount of work that can be categorized under the umbrella of “requirement-driven” machine learning, this is the first paper which explicitly advocates for a general model development methodology in which the requirement definition is a first class citizen like the other phases in the pipeline. We believe that this corresponds to a more structured approach to machine learning model development, which – as a general principle – has already been advocated in [19,49,61].