Modeling the impact of noise and uncertain operational environment on software release decisions considering testing coverage-based SRGM

Abstract

The operational environment of software differs from the debugging environment. Therefore, the study explores the impact of irregular consumption by diverse users on the development cost of software, reliability of software, and release decisions. To accomplish this, a release model has been formulated considering logistic testing coverage-based reliability growth model built as a stochastic process including the error generation. The stochastic nature has been captured by considering the noise factor due to irregular fluctuations occurring during testing while usage uncertainty has been captured by introducing a constant parameter in the cost function of the operational phase.

It is assumed that the testing phase cost is affected by the noise and the operation phase cost is affected by the severity of noise which is the result of uncertain usage by users. The model was evaluated against a real failure dataset. The release model creates a trade-off between software development cost, release timing, and reliability aspirations. This study contributes to software reliability literature and provides insights to practitioners to make software release decisions. The sensitivity analysis results give information about various aspects during the operational phase that affect the overall development cost.

Keywords

Testing coverage noise-based SRGM error generation operational phase release planning environmental factor sensitivity analysis

1. Introduction

In light of the revolution of emerging technologies, firms dealing with software development are feeling greater strain internationally to provide novel technologies that represent significant breakthroughs and give an edge over others. These evolving businesses face challenges to retain their current market share and draw new clients [1, 2]. The dependence of mankind on various fields like medicine, transportation, education, and tourism makes it important for software firms to develop products with higher reliability levels so that there is no failure during execution. Such failures can cause loss of property, and money and in critical cases, it can lead to loss of life. However, due to complicated code structures, it is not realistic to eradicate all defects during testing but it can be minimized. Errors can be introduced into the program’s code at any time throughout the development process [3].

Consumers choose long-lasting, high-performing products. Organizations prioritize developing reliable applications to prevent costly failures, safety concerns, and legal fines. When developing software, it is important to consider the frequency of failures, estimate testing effort, cease testing (testing stop time), and release timing [4, 5]. The continuous maintenance effort for a software application must be anticipated and estimated. This helps management plan and schedule activities accordingly. Some examples of this include how long the warranty should last once the software is released in the market, when the product should be released into the market, the number and severity of defects that can be expected, the number of software professionals (practitioners) required to support the product for development and maintenance [6].

The Software Reliability Growth Model (SRGM) is a mathematical instrument for Software Engineering that evaluates software, develops test plans and schedules, and monitors changes in reliability effectiveness [7]. Over the last two decades, many SRGMs have demonstrated that the connection between testing time and fault removal is either exponential, S-shaped or a combination of the two [7, 8, 9, 10, 11, 12]. Various sorts of errors in software need distinct methodologies and testing efforts to be eliminated [13, 14].

Several defects are identified and corrected to achieve the reliability goal for the software during the testing process and then it is released to the market. Customers uncover some defects during usage and report them to the developer, and then the team works on the existing code to remove the pain points and add some additional functionalities and then releases an improved version of the software. This version incorporates the solutions to the issues encountered by users while using the previous release and it also incorporates the latest technological developments that are in high demand. In this situation, the defects remaining in the system are viewed as a stochastic process [15].

Several researchers have worked on this concept and developed SRGMs using stochastic processes to incorporate irregular fluctuations. .Shigeru and Akio [16] proposed a basic SRGM to assess the detection and correction of faults leading to growth in the reliability of the software. The model used the Stochastic Differential Equation (SDE) of Itô type to derive the various standards related to using the stochastic process’s probability distribution. Later, they came up with a customizable SDE Model to describe the process of improving the reliability of software by detecting and removing the faults in an environment where different teams are involved in planning, analyzing, coding, and testing [17]. This type of environment with decentralized teams to work on various activities is known as the distributed development environment. Instead of the Non-Homogenous Poisson Process (NHPP), .Lee, and Kim [18] used SDEs to portray a per-fault detection process consisting of erratic variations, and they took into account the fault detection process dependent on testing duration $t$ .

The work presented in this manuscript employs SDEs to depict the failure behavior of software demonstrating irregular fluctuations. This research reflects the implications of actual situations influencing the testing process, like tester’s knowledge and expertise affect the rate at which error is identified in the code after the failure occurs and then its removal from the software, the portion of code tested also influences the process of removing faults, the intricacy of code could generate additional errors in the software system during debugging process which eventually affects the rate at which error is identified in the code after the failure occurs and then its removal from the software and irregular fluctuations known as white noise affects the fault removal rate [19]. To tackle this, an SRGM that accounts for the amount of code covered during testing at time $t$ , irregular fluctuations, and error generated is developed. This helps to understand the fault identification and its elimination from the software. This article expresses the process of fault detection and removal using the logistic testing coverage function [20, 21].

Rapid technological advancements necessitate frequent software upgrades. These upgrades foster a long-term commercial connection between clients and developers. The developer releases the software in the market with time by upgrading the software in each release as per the requirements of the users. This research aims to reduce software development costs by considering both external and internal environmental elements throughout the product’s life cycle, including testing coverage. The operational environment of the software life cycle differs from its debugging environment because of the existence of uncertainty during its use by diverse users. We will investigate the impact of changes due to irregular fluctuations in software testing, the severity of the irregular fluctuation during the operational phase because of diverse users, and fault elimination costs on software release [22, 23].

To accomplish the aforementioned research goals, we will create a mathematical representation that depicts the failure process while taking into account the conditions listed below:

i.
error generation phenomenon
ii.
percentage of code covered during testing (during the process of identifying a fault in software during testing the part of the code is considered for testing instead of testing the complete code at once.)
iii.
unusual variations present during testing

Further, the suggested mathematical model is utilized in an optimization model that is used for managing the entry of the software system into the market for users while aiming to reduce the overall development costs incurred during the process. This helps to determine the optimal testing periods for developers in the above-mentioned conditions [14, 24, 25, 26]. The optimization model consists of three cost components related to testing (software is in the hands of experts to identify and remove the faults) and operational phases (period after release of the product, here the software is in the hands of diverse types of users).

The conditions during the operating environment (when software has been released into the market and is extensively used by the users) are different from the debugging environment (the team of testers regressively tests the software to remove the maximum number of faults from the software). To address this difference, a constant parameter $k$ is introduced into the fixed cost part of the optimization model, which represents the severity of unanticipated irregular variations in the operating environment after release. [27, 28, 29, 30]. This parameter takes into account the uncertainty of unlimited and diverse usage of the software by the customers. We assume a proportionate link between testing and operational phase periods. To attain the objectives of the study the optimization model is solved using Genetic Algorithm (GA) in MATLAB software [31]. It is a soft computing technique that is widely employed by software practitioners and researchers to solve such dilemma problems and create a tradeoff between requirements and available resources [31, 32, 33, 34, 35].

The research paper consists of five sections. 1^st section i.e. introduction section is going on. After that, we have discussed the past developments of SDE in software reliability literature. In Section 3 the SRGM and Optimization model is formulated for the software with multiple releases while considering the parameters affecting the fault removal process. In Section 4 the model developed in the section is fitted on the failure dataset of tandem computers. This dataset presents the faults observed in the computer with testing. This helps in validating the model for the software fault datasets. Further in this section, the influence of random changes in the severity of fluctuations is investigated. In the last section of the manuscript, the conclusions of the study are presented along with that the prospects of the concepts have also been discussed.
2. Literature survey

This section discusses the past developments related to the concepts utilized in the study. These studies assist in comprehending the necessity of analyzing current volatility and identifying research gaps.

Gokhale, Philip [36] developed a unified NHPP SRGM where the fault removal process was based on the testing coverage. Later, exponential, Weibull, and logistic testing coverage-based SRGMs were also proposed by them [37]. Pham and Zhang [20] proposed an SRGM that represents a failure curve in the form of $S$ -Shape and its FDR is based on the portion of code covered for testing. This is known as testing coverage (TC).

INOUE and YAMADA [38] gave a 1-D TC-based SRGM assuming that it is affected by the testing time and observed that coverage is highly impacted by the skills of the testers. Li and Pham [39] proposed TC-based SRGM by representing fault detection rate (FDR) as TC and validated the model on real-life data under the uncertainty in the operating environment and later Presented an SRGM using testing coverage and considered fault removal efficiency and error generation [39]. Aggarwal and Kumar [40] used three TCF were used to generate multi-release SRGM namely exponential, logistic, and S-shaped functions. Bibyan and Anand [41] proposed a unified SRGM using different testing coverage functions and estimated parameters using a neural network.

To capture the irregular fluctuation as noise, SDE-based SRGM gas has been used by several researchers in the past. The first attempt to use SDE was made by Yamada, and Kimura [42]. According to Shigeru and Akio [16], the faults that could not be removed from the program may be modelled as a stochastic process utilizing Itô-type SDE to generate SRGMs with different failure curves such as exponential failure curve, delayed S-shaped failure curve, and inflected S-shaped failure curve. Shyur [43] incorporated the parameters for the imperfect debugging environment and change in FDR in SDE-based SRGM. The change in FDR is also termed a change point.

Tamura and Yamada [17] Suggested flexible SDE-based SRGM for open-source software. The model discussed by them explains different patterns for reliability growth while considering the impact of different testing efforts employed to achieve reliability goals for the software. Kapur, Anand [44] proposed a generalized flexible SDE-based SRGM and extended the work by studying three types of faults namely simple, hand, and complex using SDE on Itô type [45]. The model discussed by them could explain the reliability behavior of software showing $S$ -shaped curve and exponential growth. Singh and Kapur [46] proposed an SRGM for an open-source software using Itô type SDE and compared the results with the existing models. Fang and Yeh [47] developed SRGM using existing SDE models with confidence intervals to help the developers find the optimal release time.

Kapur, Anand [48] created a unified framework to develop SRGM using SDE. The authors proposed an SRGM that determines the number of faults removed from the considering that the process of removing faults is affected by the efforts employed during the testing process. Pham and Pham [49] incorporated the concept of environmental factors into their SDE-based SRGM. The SRGM model describes FDR using a Weibull distribution incorporating the influence of various environmental conditions (related to various phases of software development) quality of the software. Chatterjee and Chaudhuri [50] presented the use of SDE-based SRGM in the release planning model.

The suggested method determines release time by uniting metaheuristic algorithms and considering the budget affected by randomness under an intuitionistic fuzzy environment. Simulation findings show that the suggested strategy is successful at improving software dependability while minimizing costs. Later, they suggested an SDE-based SRGM in which fault detection is connected to the severity of fault incidence and fault correction is tied to the necessity of fault correction. The problem developed helps to create a tradeoff between the cost incurred in development and the reliability achieved after testing. The cost constraint consisted of coefficients given by interval type-2 fuzzy numbers. The type-2 fuzzy numbers are more accurate to reflect the variability of specifications given for software development [51]. Zheng, Yang [52] presented an SRGM incorporating fluctuation.

The SDE-based SRGM represented the disruption in the continuity due to uncertain random events. Bibyan and Anand [19] incorporated the concept of multi-release of the software by proposing SDE-based SRGM using different testing coverage functions. Later, the model was extended by incorporating an error generation parameter and the change point concept. A release model was suggested to capture the effect of irregular fluctuations on the cost used for software development and to observe its effect on the time decided for the release of the product [53, 54, 55, 56]. Table 1 summarizes SDE-based Models and our proposed work.

Table 1
SDE literature

Year	Contribution
2003	According to Shigeru and Akio [16], the faults that could not be removed from the program may be modelled as a stochastic process utilizing Itô-type SDE to generate SRGMs with different failure curves such as exponential failure curve, delayed S-shaped failure curve, and inflected S-shaped failure curve.
2003	Shyur [43] incorporated the parameters for the imperfect debugging environment and change in FDR in SDE-based SRGM. The change in FDR is also termed a change point.
2006	Tamura and Yamada [17] Suggested flexible SDE-based SRGM for open-source software. The model discussed by them explains different patterns for reliability growth while considering the impact of different testing efforts employed to achieve reliability goals for the software.
2007	Kapur, Anand [44] proposed a generalized flexible SDE-based SRGM and extended the work by studying three types of faults namely simple, hand, and complex using SDE on Itô type. The model discussed by them could explain the reliability behavior of software showing $S$ -shaped curve and exponential growth.
2010	Singh, and Kapur [46] proposed an SRGM for an open-source software using Itô type SDE and compared the results with the existing models.
2011	Fang and Yeh [47] developed SRGM using existing SDE models with confidence intervals to help the developers find the optimal release time.
2012	Kapur, Anand [48] created a unified framework to develop SRGM using SDE. The authors proposed an SRGM that determines the number of faults removed from the considering that the process of removing faults is affected by the efforts employed during the testing process.
2019	Pham and Pham [49] incorporated the concept of environmental factors into their SDE-based SRGM. The SRGM model describes FDR using a Weibull distribution incorporating the influence of various environmental conditions (related to various phases of software development) quality of the software.
2020	Chatterjee, and Chaudhuri [50] presented the use of SDE-based SRGM in the release planning model.
Proposed work	In this work, the impact of random fluctuations (white noise) due to uncertain events happening during the fault removal process on the ideal time to release the software into the market is analysed. A logistic testing coverage-based SRGM is used, which accounts for the variations due to noise. It is modelled using SDE.

In this work, we will look at the impact of random fluctuations (white noise) due to uncertain events happening during the fault removal process on the ideal time to release the last release of multi-release software into the market. A logistic testing coverage-based SRGM is used, which accounts for the variations due to noise. It is modelled using SDE. The proposed SRGM also evaluates the influence of errors that are generated in the software when a particular fault is removed during testing on reliability achieved by the software. The existence of irregular fluctuations during testing, as well as their severity throughout the operating phase, affect the fixed cost parameter. Additionally, the sensitivity of the severity of fluctuation in this model has been investigated.

3. Model development

Here, under this section, we will develop an SRGM considering logistic testing coverage function incorporating irregular fluctuations during the testing phase and error generation phenomenon and further we will discuss a release model to determine the optimal testing duration that takes into account the impact of uncertain usage environment. The severity of the irregular fluctuations tends to change during the operational phase due to the diverse range of users.

3.1 SRGM development using SDE

Let’s start with a discussion of the notations and assumptions needed to create the SRGM.

3.1.1 Notations

$S\left(t\right)$ :	The random variable representing faults removed in time $t$ .
$a\left(t\right)$ :	It represents the time-dependent fault content in software at time $t$ .
$m\left(t\right)=E\left({S\left(t\right)}\right)$ :	It represents the cumulative number of faults detected/removed up to time $t$ .
$a$ :	It represents the constant fault content of software which is not affected by the testing time.
$b(t)$ :	Rate of detecting faults from the software.
$\beta$ :	A constant parameter of the logistic function.
$\theta$ :	Error generation probability
$\sigma$ :	Magnitude of irregular fluctuations.
$W\left(t\right)$ :	1-D Weiner process

3.1.2 Assumptions

i.
The FDR is modeled using SDE.
ii.
The FDR is affected by the leftover faults i.e. the faults that have not been removed from software till that time.
iii.
The software may fail or cause some error in functioning during execution due to the faults already present in the software.
iv.
The faults present in the software are mutually dependent i.e. action taken on a fault affects other faults also.
v.
The debugging process is not perfect i.e. some additional defects may get introduced while debugging
vi.
The number of faults present in the software at a particular time changes due to the addition of errors during debugging and the removal of additional faults. The faults getting added are proportional to the amount of time spent testing it.
vii.
The testing and operational phases differently affect the reliability of the software. The uneven variations (during the testing phase) and the uncertain usage of software (during the operational phase) have an impact on the program’s reliability.
viii.
The FDR is expressed in terms of testing coverage and is mathematically given as $\frac{c^{\prime}\left(t\right)}{1-c\left(t\right)}$ .

Using the abovementioned notations, the corresponding differential equation to depict the fault removal phenomenon by time $t$ is given as:

$\displaystyle\frac{dS\left(t\right)}{dt}=b\left(t\right)\left[{a\left(t\right)% -m\left(t\right)}\right]$ (1)

Equation (1) can also be represented in terms of testing coverage: (By using assumption (viii))

$\displaystyle\frac{dS\left(t\right)}{dt}=\frac{c^{\prime}\left(t\right)}{1-c% \left(t\right)}\left[{a\left(t\right)-S\left(t\right)}\right],$ (2)

where, $c^{\prime}(t)=\frac{dc}{dt}$ ,

It has been assumed in this study, that the error generation is dependent on testing time (assumption vi) $a\left(t\right)=a+\theta S\left(t\right)$ . So, Eq. (2) can be rewritten as:

$\displaystyle\frac{dS\left(t\right)}{dt}=\left[{\frac{c^{\prime}\left(t\right)% }{1-c\left(t\right)}+\textit{Noise}}\right]\left[{a+\theta S\left(t\right)-S% \left(t\right)}\right],$ (3)

The standard Gaussian white noise has been considered to represent the irregular fluctuations and is denoted as $\varphi\left(t\right)$ with a $\sigma$ as a constant that measures the magnitude of the fluctuations. So,

$\displaystyle b\left(t\right)=\frac{c^{\prime}\left(t\right)}{1-c\left(t\right% )}+\sigma\varphi\left(t\right),$ (4)

Hence, Eq. (3) can be written using Eq. (3) as

$\displaystyle\frac{dS\left(t\right)}{dt}=\left[{\frac{c^{\prime}\left(t\right)% }{1-c\left(t\right)}+\sigma\varphi\left(t\right)}\right]\left[{a-\left({1-% \theta}\right)S\left(t\right)}\right],$ (5)

And,

$\displaystyle\frac{dS\left(t\right)}{dt}=\left[{\left({1-\theta}\right)\frac{c% ^{\prime}\left(t\right)}{1-c\left(t\right)}+\sigma\varphi\left(t\right)}\right% ]\left[{a^{\prime}-S\left(t\right)}\right],\ \textit{where}\ a^{\prime}=\frac{% a}{1-\theta}$ (6)

Equation (6) is written as SDE of an Itô type and is given by Eq. (7):

$\displaystyle dS\left(t\right)=\left[{\left({1-\theta}\right)\frac{c^{\prime}% \left(t\right)}{1-c\left(t\right)}-\frac{1}{2}\sigma^{2}}\right]\left[{a^{% \prime}-S\left(t\right)}\right]dt+\sigma\left[{a^{\prime}-S\left(t\right)}% \right]dW(t),$ (7)

where, $W\left(t\right)$ represents the 1-D Wiener process, which integrates white noise $\gamma(t)$ over time $t$ . Furthermore, the Wiener process ( $w\left(t\right)$ ) is also a Gaussian process. This process has the below-mentioned properties:

i.
Probability $\left[{w\left(0\right)=0}\right]=1$
ii.
Expectation $\left[{w\left(t\right)}\right]=0$
iii.
Expectation $\left[w(t)w(t^{\prime})\right]=\min[t,t^{\prime}]$

Now, the Itô solution has been applied to Eq. (7) utilizing conditions $t=$ 0, $S\left(0\right)=$ 0 to solve the SDE and obtain the equation for the number of faults removed at time $t$ under existing variations (Eq. (8)).

$\displaystyle S(t)=\frac{a}{1-\rho}\left[1-\exp\left(-(1-\theta)\int_{0}^{t}% \frac{c^{\prime}(x)}{1-c(x)}dx+\sigma W(t)\right)\right],$ (8)

The expectation of Eq. (8) has been taken:

$\displaystyle E(S(t))=m(t)=\frac{a}{1-\theta}\left[1-\exp\left(-(1-\theta)\int% _{0}^{t}\frac{c^{\prime}(x)}{1-c(x)}dx+\frac{\sigma^{2}t}{2}\right)\right]$ (9)

Now considering the Logistic testing coverage function (assumptions ii. iii and iv)

$\displaystyle\frac{c^{\prime}\left(x\right)}{1-c^{\prime}\left(x\right)}=\frac% {b}{1+\beta e^{-bt}},$ (10)

On putting the Eq. (10) in Eq. (9), We have

$\displaystyle m\left(t\right)=\frac{a}{1-\theta}\left[{1-\left({\frac{1+\beta}% {1+\beta e^{-bt}}}\right)^{\left({1-\theta}\right)}\exp\left({-b\left({1-% \theta}\right)t+\frac{\sigma^{2}t}{2}}\right)}\right],$ (11)

The final Eq. (11) obtained after solving Eq. (10) using Eq. (9) presents the cumulative number of defects a tester was able to remove within time $t$ . This equation is termed as Mean Value Function (MVF) to assess the reliability growth of software systems with Logistic Testing Coverage, error generation, and irregular fluctuations. MVF can be defined as a statistical model used to capture the failure behavior of software applications. It predicts the mean value of software reliability over time. It helps testers and developers to track the reliability of software and aid them in making release decisions. This equation (Eq. (11) is further used for evaluating the reliability of the multiple releases of the software under the same conditions. The expressions of the various releases are given in Eqs (12)–(14).

For release 1 the MVF is

$\displaystyle m_{1}\left(t\right)=\left[{\frac{a_{1}}{1-\theta_{1}}}\right]D_{% 1}\left(t\right),$ (12)

where

$\displaystyle D_{1}\left(t\right)=\left[{1-\left({\frac{1+\beta_{1}}{1+\beta_{% 1}e^{-b_{1}t}}}\right)^{\left({1-\theta_{1}}\right)}\exp\left({-b_{1}\left({1-% \theta_{1}}\right)t+\frac{\sigma_{1}^{2}t}{2}}\right)}\right],\ \text{and}\ 0<% t<t_{1}.$

And $t_{1}$ is the release time of the first release.

For release 2 the MVF is

$\displaystyle m_{2}\left(t\right)=\left[{\frac{a_{2}}{1-\theta_{2}}+\frac{a_{1% }}{1-\theta_{1}}(1-D_{1}\left({t_{1}}\right)))}\right]D_{2}\left({t-t_{1}}% \right),$ (13)

where

$\displaystyle D_{2}\left(t\right)=\left[{1-\left({\frac{1+\beta_{2}}{1+\beta_{% 2}e^{-b_{2}t}}}\right)^{\left({1-\theta_{2}}\right)}\exp\left({-b_{2}\left({1-% \theta_{2}}\right)t+\frac{\sigma_{2}^{2}t}{2}}\right)}\right],\ \text{and}\ t_% {1}<t<t_{2}.$

For $n^{\text{th}}$ release the expression for $m\left(t\right)$ is given by as:

$\displaystyle m_{n}\left(t\right)=\left[\frac{a_{n}}{1-\theta_{n}}+\frac{a_{n-% 1}}{1-\theta_{n-1}}(1-D_{n-1}\left({t_{n-1}}\right))+\frac{a_{n-2}}{1-\theta_{% n-2}}(1-D_{n-2}\left({t_{n-2})}\right)\right.(1-D_{n-1}\left({t_{n-1}}\right))% +\ldots+\frac{a_{n-\left({n-1}\right)}}{1-\theta_{n-\left({n-1}\right)}}(1-D_{% n-1}\left({t_{n-1}}\right))\left.\phantom{\hskip-5.690551pt\frac{1}{2}}\ldots% \left(1-D_{n-\left({n-1}\right)}\left({t_{n-\left({n-1}\right)}}\right)\right)% \right]D_{n}\left({t-t_{n-1}}\right),$ (14)

where

$\displaystyle D_{n}\left(t\right)=\left[{1-\left({\frac{1+\beta_{n}}{1+\beta_{% n}e^{-b_{n}t}}}\right)^{\left({1-\theta_{n}}\right)}\exp\left({-b_{n}\left({1-% \theta_{n}}\right)t+\frac{\sigma_{n}^{2}t}{2}}\right)}\right],\ \text{and}\ t_% {n-1}<t<t_{n}.$

Equation (14) presents $m\left(t\right)$ for the $n^{\text{th}}$ release. The final value obtained after putting replaced values gives the faults that have been eradicated up to testing time $t$ . The conditions like irregular fluctuations, error generation, and logistic testing coverage influence the fault removal phenomenon which in turn influences the reliability that testers could attain after testing the software for a certain period. All these things eventually affect the decisions taken by software practitioners and the management pertaining to the release of software into the market.
3.2 Release planning model

The SRGM developed in the previous section will be considered to find the best time to bring the $n^{\text{th}}$ release of the software into the market under the constraint of reliability to be attained. The release management problem formulated in the form of an optimization problem (P1) to manage the release of software has the objective of reducing software development costs while meeting the program’s reliability criteria. Before defining the release problem, we will discuss a few assumptions of the release model.

1.
Testing cost is linearly related to the number of faults removed from the software.
2.
The cost of removing faults during operations is linearly related to the number of faults latent in the software.
3.
The irregular fluctuations are affecting testing costs by constant severity.

P1 defines the release problem in words:

$\displaystyle\left.\begin{array}[]{llc}\textit{Minimize}&&\textit{Total % Software Development Cost}\\ &&s.t.\\ &&\textit{Reliability Constraint}\end{array}\right\}\ \ \ \ \text{{P1}}$

The total cost of developing the $n^{\text{th}}$ release of software with added functionalities is represented by Eq. (15).

$\displaystyle\textit{Cost}_{n}\left(T\right)=\textit{Cost}_{n_{1}}m_{n}\left(T% \right)+Cost_{n_{2}}\left({a_{n}^{\ast}-m_{n}\left(T\right)}\right)+\frac{% \textit{Cost}_{n_{3}}T}{1-\left({1-\sigma_{n}}\right)^{k}\left({1-\theta_{n}}% \right)},$ (15)

where the notations carry the following meanings:

Cost_n : The cost that is expected to be incurred during the software development process that will involve all the software components.

$\textit{Cost}_{n_{1}}$ : The unit cost of removing faults during testing for a particular release.

$\textit{Cost}_{n_{2}}$ : The unit of removing faults during the operational phase for a particular release.

$\textit{Cost}_{n_{3}}$ : The fixed cost of Testing the code developed for that release.

$k$ : The severity of irregular fluctuations or impact of noise on the fixed cost of testing.

The cost model developed in this sub-section shows that it consists of three cost components (fixed cost of testing, testing cost with time when faults are removed and operational cost when the software is in the hands of the user). The operating phase cost is referred to as the cost of penalty for incurring errors during the execution phase. The errors in usage period create a negative impact on the product’s image. The fixed cost of testing is presented in the third cost component of Eq. (15). The fixed of testing includes the cost associated with providing the necessary resources. It is affected by the errors generated during debugging and the irregular fluctuations during the testing phase. The factor $k$ in the model represents the severity of irregular fluctuations.

In $\frac{\textit{Cost}_{n_{3}}T}{1-\left({1-\sigma_{n}}\right)^{k}\left({1-\theta% _{n}}\right)}$ , if $\sigma_{n}\to 0$ and $\theta_{n}\to 0$ then the overall development cost $\textit{Cost}_{n_{3}}T\to\infty$ . This factor also eventually affects the cost of testing. This means to reduce the irregular fluctuations and also to reduce the errors added during debugging then the developers need to employ additional resources, which will eventually increase the development cost. The resources can comprise of time required to remove the faults, employing more skilled people, etc.

The aim of the problem is to achieve the set reliability goal for the software using a minimum amount. The reliability should be greater than the required reliability. Mathematically, this constraint is given by Eq. (16).

$\displaystyle\textit{Reliability}_{n}\left({x{|}T}\right)\geqslant R_{0},$ (16)

Where $\textit{Reliability}_{n}(x|T)$ represents the reliability of software after testing it for $T$ duration during the $n^{\text{th}}$ release. It is represented using Eq. (17). In this equation $R_{0}$ represents the reliability attainment level for the software which has been set in advance by the developers. In this equation, $x$ is used to show the small change in testing time.

$\displaystyle\textit{Reliability}_{n}\left({x|T}\right)=\exp\left({-\left({m_{% n}\left({T+x}\right)-m_{n}\left(T\right)}\right)}\right),$ (17)

The problem P1 defined in words is replaced by Eqs (15) and (16)

$\displaystyle\left.\begin{array}[]{llc}\textit{Minimize}&&\textit{Cost}_{n_{1}% }m_{n}\left(T\right)+\textit{Cost}_{n_{2}}\left({a_{n}^{\ast}-m_{n}\left(T% \right)}\right)+\frac{\textit{Cost}_{n_{3}}T}{1-\left({1-\sigma_{n}}\right)^{k% }\left({1-\theta_{n}}\right)}\\ &&s.t.\\ &&\textit{Reliability}_{n}(x|T)\geqslant R_{0}\end{array}\right\}\ \ \ \ \text% {\text{P2}}$

where $m(T)$ is provided by Eq. (13), $\textit{Reliability}_{n}{(x|T})$ is provided by Eq. (17) and $T$ represents the best time to launch the software into the market. The problem P2 is solved in MATLAB to determine $T$ .
4. Results and discussions

Cost_n :	The cost that is expected to be incurred during the software development process that will involve all the software components.
$\textit{Cost}_{n_{1}}$ :	The unit cost of removing faults during testing for a particular release.
$\textit{Cost}_{n_{2}}$ :	The unit of removing faults during the operational phase for a particular release.
$\textit{Cost}_{n_{3}}$ :	The fixed cost of Testing the code developed for that release.
$k$ :	The severity of irregular fluctuations or impact of noise on the fixed cost of testing.

4.1 SRGM validation

Here, the SRGM proposed in Section 3 is validated using a failure dataset of faults observed in the computer with testing time to estimate the value for the parameters of the logistic testing coverage-based SRGM given in Eq. (11). The dataset used for validation purposes is of Tandem Computers (Wood (1996)). The process is performed using a statistical tool called SPSS. The description of the failure data can be referred from Aggarwal, Anand [54]. There are four releases in the dataset. Release 1 was tested for 20 weeks and 100 faults were observed. Release 2 was tested for 19 weeks and 120 faults were observed. Release 3 was tested for 12 weeks and 61 faults were observed. Release 4 is tested for 19 weeks and 42 faults are observed. The results of parameter estimation are provided in Table 2.

Table 2
Estimated parameters

Releases	$a$	$b$	$\beta$	$\theta$	$\sigma$	$R^{2}$
Release 1	113	0.0324	0.863	0.241	0.00247	0.997
Release 2	121.6	0.0516	2.324	0.005	0.00010	0.996
Release 3	61.4	0.1397	21.327	0.152	0.00199	0.998
Release 4	44.2	0.05787	1.9438	0.017	0.009709	0.998

From Table 2 we observe the little presence of noise during the debugging process. The findings reveal that the created model accurately captures the software’s failure behavior. The data from the fourth release are utilized to solve P2 and determine the best release time under given constraints assuming that the prior version is available in the market. Problem P2 seeks to minimize the cost involved in the software development process while maintaining the needed reliability level of software, taking into account other parameters that significantly influence the debugging process. The last column of the table shows that the model fits the data for all releases. As $R^{2}$ values for all four releases is greater than 99%. This implies that all the parameters considered (error generation, irregular fluctuations during testing, testing coverage, fault dependency) to capture the failure phenomenon can explain the 99% of failures for all releases.

4.2 Release planning

After validating the SRGM, the next step is to determine the optimal variables. The P2 problem is discussed in Section 3.2 and is answered using GA in MATLAB. The GA is commonly employed in the field to find the appropriate release timing [32, 57, 58]. Now, to solve P2, we assume the cost values as Cost ${}_{41}=$ 10, Cost ${}_{42}=$ 15, and Cost ${}_{43}=$ 5 for the 4^th release, and the reliability goal is set at 90%. The optimization problem for the 4^th release is given by P3. Replacing the assumed values in P3 it is solved in MATLAB using GA algorithm.

$\displaystyle\left.\begin{array}[]{llc}\textit{Minimize}&&\textit{Cost}_{41}m_% {4}\left(T\right)+\textit{Cost}_{42}\left({a_{n}^{\ast}-m_{4}\left(T\right)}% \right)+\frac{\textit{Cost}_{43}T}{1-\left({1-\sigma_{4}}\right)^{k}\left({1-% \theta_{4}}\right)}\\ &&s.t.\\ &&\text{exp}\left({-\left({m_{4}\left({T+x}\right)-m_{4}\left(T\right)}\right)% }\right)\geqslant 0.90\\ &&\textit{where}\ T>T_{n-1}\\ \end{array}\right\}\ \ \ \ \text{\text{P3}}$

$m_{4}(T)$ is given by Eq. (11) and $a_{4}^{\ast}$ is given by Eq. (15) for $n=4$ .

On solving P3 we obtained 110 weeks as the optimal time for the fourth release. This implies that software needs to be under testing for 110 weeks to achieve 90% reliability. This target has been achieved by utilizing 1649 units of resources. If the software is tested even after 110 weeks we will be able to achieve higher reliability levels. Figures 1 and 2 depict the development cost and reliability curves respectively for the software.

Table 3
Sensitivity results of parameter $k$

$k$	Fixed cost
0.1	11046.42
0.2	11037.58
0.5	11011.15
0.7	10993.62
0.9	10976.14
1.2	10950.05

Figure 1.

Cost curve.

Figure 2.

Reliability curve.

From Fig. 1 we can observe that initially, the cost required is very high but as the testing progresses the cost keeps on going down. The requirement of 90% reliability is fulfilled after 110 weeks of testing. Similarly, after observing Fig. 2 we conclude that 100% reliability for the software can be obtained if testing continues for prolonged durations. In practical situations, the testing cannot be continued for infinite duration because with time the cost of identifying fewer faults and removing them from the software takes more effort, time, and cost.

4.3 Sensitivity analysis of

k

From the cost model, it is clear that the parameter $k$ which represents the severity of irregular fluctuations impacts the fixed cost involved in software development. Hence, we see the comparative variation in fixed cost owing to a 10% alteration in the value of $k$ . Table 3 presents the value of fixed cost for a particular value of $k$ . As the severity of irregular fluctuation increases the cost decreases i.e. the same level of reliability is achieved by investing less in the process. The cost reduction can be attributed to the learning that is gained by the tester with each fault.

5. Conclusions and scope for future research

The study investigated the effect of unbalanced fluctuations on the optimum release time and software development costs. An SRGM has been created for this purpose, which is based on the logistic testing coverage-based SRGM that includes the error generation phenomena. The model was verified using a real-life defect dataset from Tandem computers. The $R^{2}$ for the releases (Table 1) was 0.99 which implies that the factors or parameters considered can explain 99% of the failure behavior of the software. Consequently, the model is an excellent fit for the failure data. Further, in the paper, a release model has been developed that captures the impact or severity of irregular fluctuations by including it as a constant parameter in the fixed cost. It is observed that as the impact of irregular fluctuation increases the cost decreases i.e. the same level of reliability is achieved by investing less in the process. The cost reduction can be attributed to the learning that is gained by the tester with each fault.

Footnotes

Acknowledgments

This research was supported by an FRP grant received from the Institution of Eminence, University of Delhi, India (Ref No. IoE/2023-24/12/FRP).

References

Zhuravel

Boltianska

. Integrated approach to ensuring the reliability of complex systems. In: Current issues, achievements and prospects of Science and education: Abstracts of XII International Scientific and Practical Conference. Athens, Greece: 2021; pp. 231-233.

Royce

. Managing the Development of Large Software Systems. MIT Press Direct. 2021. doi: 10.7551/mitpress/12274.003.0035.

Zhai

Martínez,

Beltran,

Martínez

. Decision support systems for agriculture 40: Survey and challenges. Computers Electronics in Agriculture. 2020; 170: p. 105256.

Chen

Wei

. Towards people-centric smart city development: Investigating the citizens’ preferences and perceptions about smart-city services in Taiwan. Sustainable Cities and Society. 2021; 67: p. 102691.

Schomakers

Biermann

Ziefle

. Users’ preferences for smart home automation–investigating aspects of privacy and trust. Telematics and Informatics. 2021; 64: p. 101689.

Al-Saqqa

Sawalha

AbdelNabi

. Agile software development: Methodologies and trends. International Journal of Interactive Mobile Technologies. 2020 Nov 1; 14(10).

Kapur

Kumar

Garg

. Contributions to hardware and software reliability. World Scientific, 1999.

Goel

Okumoto

. Time-dependent error-detection rate model for software reliability and other performance measures. IEEE transactions on Reliability. 1979 Aug; 28(3): 206-211.

Kapur

Garg

. A software reliability growth model for an error-removal phenomenon. Software Engineering Journal. 1992; 7(4): 291-294.

10.

Ohba

. Software reliability analysis models. IBM Journal of Research and Development. 1984; 28(4): 428-443.

11.

Ohba

. Inflection S-shaped software reliability growth model, in Stochastic models in reliability theory. 1984; Springer. p. 144-162.

12.

Yamada

Ohba

Osaki

. S-shaped software reliability growth models and their applications. IEEE Transactions on Reliability, 1984; 33(4): 289-292.

13.

Thimbleby

. The problem isn’t Excel, it’s unprofessional software engineering. BMJ, 2020; 371.

14.

Verma

Anand

Kapur

Aggarwal

. Unified framework to assess software reliability and determine optimal release time in presence of fault reduction factor, error generation and fault removal efficiency. International Journal of System Assurance Engineering and Management. 2022 Oct; 13(5): 2429-2441.

15.

Øksendal

. Stochastic differential equations, in Stochastic differential equations. 2003; Springer. pp. 65-84.

16.

Shigeru

Akio

. A stochastic differential equation model for software reliability assessment and its goodness-of-fit. International Journal of Reliability and Applications. 2003; 4(1): 1-12.

17.

Tamura

Yamada

. A flexible stochastic differential equation model in distributed development environment. European Journal of Operational Research. 2006; 168(1): 143-152.

18.

Lee

Kim

Park

. S-shaped software reliability growth models derived from stochastic differential equations. IIE transactions. 2004; 36(11): 1193-1199.

19.

Bibyan

Anand

Aggarwal

Kaur

. Multi-release software model based on testing coverage incorporating random effect (SDE). MethodsX. 2023 Jan 1; 10: 102076.

20.

Pham

Zhang

. NHPP software reliability and cost models with testing coverage. European Journal of Operational Research. 2003 Mar 1; 145(2): 443-454.

21.

Chang

Pham

Lee

Song

. A testing-coverage software reliability model with the uncertainty of operating environments. International Journal of Systems Science: Operations & Logistics. 2014 Oct 2; 1(4): 220-227.

22.

Rüßmann

Lorenz

Gerbert

Waldner

Justus

Engel

Harnisch

. Industry 40: The future of productivity and growth in manufacturing industries. Boston Consulting Group. 2015 Apr 9; 9(1): 54-89.

23.

Nunamaker

, Jr. Chen

Purdin

. Systems development in information systems research. Journal of Management Information Systems. 1990 Dec 1; 7(3): 89-106.

24.

Kapur

Pham

Aggarwal

Kaur

. Two dimensional multi-release software reliability modeling and optimal release planning. IEEE Transactions on Reliability. 2012 Jul 13; 61(3): 758-768.

25.

Chatterjee

Singh

. A NHPP based software reliability model and optimal release policy with logistic – exponential test coverage under imperfect debugging. International Journal of System Assurance Engineering and Management. 2014 Sep; 5: 399-406.

26.

Aggarwal

Nijhawan

Kapur

. A discrete SRGM for multi-release software system with imperfect debugging and related optimal release policy. In 2015 International Conference on Futuristic Trends on Computational Analysis and Knowledge Management (ABLAZE), 2015 Feb 25 (pp. 186-192). IEEE.

27.

Rinsaka

Dohi

. Optimal testing/maintenance design in a software development project. Electronics and Communications in Japan (Part III: Fundamental Electronic Science). 2006 Aug; 89(7): 1-9.

28.

Rinsaka

Dohi

. Determination of optimal warranty period in a software development project. In Advanced Reliability Modeling. 2004; 403-410.

29.

Rinsaka

Dohi

. Determining the optimal software warranty period under various operational circumstances. International Journal of Quality & Reliability Management, 2005; 22(6): 715-730.

30.

Verma

Anand

Aggarwal

. Software warranty cost optimization under imperfect debugging. International Journal of Quality & Reliability Management, 2019; 37(9/10): 1233-1257.

31.

Holland

. Genetic algorithms. Scientific American. 1992; 267(1): 66-73.

32.

Kim

Lee

Baik

. An effective approach to estimating the parameters of software reliability growth models using a real-valued genetic algorithm. Journal of Systems and Software. 2015 Apr 1; 102: 134-144.

33.

Mirjalili

. Genetic algorithm. Evolutionary algorithms and neural networks: Theory and applications. 2019: pp. 43-55.

34.

Mathew

. Genetic algorithm. Report submitted at IIT Bombay. 2012; 53.

35.

Gen

Lin

. Genetic algorithms and their applications. InSpringer handbook of engineering statistics 2023, London: Springer. pp. 635-674.

36.

Gokhale

Philip

Marinos

Trivedi

. Unification of finite failure non-homogeneous Poisson process models through test coverage. In Proceedings of ISSRE’96: 7th International Symposium on Software Reliability Engineering, 1996 Oct 30 (pp. 299-307). IEEE.

37.

Gokhale

Trivedi

. A time/structure based software reliability model. Annals of Software Engineering. 1999 Feb; 8(1): 85-121.

38.

Inoue

Yamada

. Testing-coverage dependent software reliability growth modeling. International Journal of Reliability, Quality and Safety Engineering. 2004 Dec; 11(4): 303-312.

39.

Pham

. NHPP software reliability model considering the uncertainty of operating environments with imperfect debugging and testing coverage. Applied Mathematical Modelling. 2017 Nov 1; 51: 68-85.

40.

Aggarwal

Kumar

Gupta

. Multi-release software reliability assessment: testing coverage-based approach. International Journal of Mathematics in Operational Research. 2023; 24(4): 583-594.

41.

Bibyan

Anand

Jaiswal

Aggarwal

. Software reliability testing coverage model using feed-forward back propagation neural network. International Journal of Modelling, Identification and Control. 2023; 43(2): 126-133.

42.

Yamada

Kimura

Tanaka

Osaki

. Software reliability measurement and assessment with stochastic differential equations. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences. 1994 Jan 25; 77(1): 109-116.

43.

Shyur

. A stochastic software reliability model with imperfect-debugging and change-point. Journal of Systems and Software. 2003 May 15; 66(2): 135-141.

44.

Kapur

Anand

Yadavalli

Beichelt

. A generalised software growth model using stochastic differential equation. Communication in Dependability and Quality Management Belgrade, Serbia. 2007; 34.

45.

Kapur

Anand

Yamada

Yadavalli

. Stochastic differential equation-based flexible software reliability growth model. Mathematical Problems in Engineering. 2009; 2009. Article ID 581383. doi: 10.1155/2009/581383.

46.

Singh

Kapur

Tandon

. Measuring reliability growth of open source software by applying stochastic differential equations. In 2010 Second World Congress on Software Engineering. 2010 Dec 19 (Vol. 2, pp. 115-118). IEEE.

47.

Fang

Yeh

. Confidence interval estimation of software reliability growth models derived from stochastic differential equations. In 2011 IEEE International Conference on Industrial Engineering and Engineering Management. 2011 Dec 6 (pp. 1843-1847). IEEE.

48.

Kapur

Anand

Yadav

Singh

. A unified scheme for developing software reliability growth models using stochastic differential equations. International Journal of Operational Research. 2012 Jan 1; 15(1): 48-63.

49.

Pham

. A generalized software reliability model with stochastic fault-detection rate. Annals of Operations Research. 2019 Jun 1; 277: 83-93.

50.

Chatterjee

Chaudhuri

Bhar

. Optimal release time determination in intuitionistic fuzzy environment involving randomized cost budget for SDE-based software reliability growth model. Arabian Journal for Science and Engineering. 2020 Apr; 45: 2721-2741.

51.

Chatterjee

Chaudhuri

Bhar

. Optimal release time determination via fuzzy goal programming approach for SDE-based software reliability growth model. Soft Computing. 2021 Mar; 25(5): 3545-3564.

52.

Zheng

Yang

Wang

. Open-source Software Reliability Modeling with Stochastic Impulsive Differential Equations. Eksploatacja i Niezawodność. 2023; 25(2).

53.

Bibyan

Anand

Aggarwal

Kaur

. Testing coverage-based software modeling incorporating random effect with change point and its release. Intelligent Decision Technologies. (Preprint): p. 1-3.

54.

Aggarwal

Anand

Bibyan

Verma

. Analyzing the Effects of White Noise on Software Release Planning Using SDE-Based SRGM. Fluctuation and Noise Letters. 2023: p. 2440003.

55.

Bibyan

Anand

Aggarwal

Tandon

. Stochastic Differential Equation-Based Testing Coverage SRGM by Using ANN Approach. In International Conference On Innovative Computing And Communication 2023 Feb 17 (pp. 397-410). Singapore: Springer Nature Singapore. doi: 10.1007/978-981-99-3010-4_34.

56.

Bibyan

Anand

Aggarwal

Tandon

. Multi-release testing coverage-based SRGM considering error generation and change-point incorporating the random effect. International Journal of System Assurance Engineering and Management. 2023 Oct; 14(5): 1877-1887.

57.

Maarof

Ismail

Fadzli

. Optimization of SARIMA model using genetic algorithm method in forecasting Singapore tourist arrivals to Malaysia. Appl Math Sci. 2014; 8(169-172): 8481-8491.

58.

Abualigah

Khader

Al-Betar

. Unsupervised feature selection technique based on genetic algorithm for improving the text clustering. In 2016 7th international conference on computer science and information technology (CSIT). 2016 Jul 13 (pp. 1-6). IEEE.

Modeling the impact of noise and uncertain operational environment on software release decisions considering testing coverage-based SRGM

Abstract

Keywords

1. Introduction

Table 1 SDE literature

3.1 SRGM development using SDE

3.1.1 Notations

3.1.2 Assumptions

4.1 SRGM validation

Table 2 Estimated parameters

Table 3 Sensitivity results of parameter k

5. Conclusions and scope for future research

Footnotes

Acknowledgments

References

Table 1
SDE literature

Table 2
Estimated parameters

Table 3
Sensitivity results of parameter $k$