Testing coverage-based software modeling incorporating random effect with change point and its release

Abstract

The use of computers has been increasingly prevalent in our social lives in recent years. As a result, software engineers must create trustworthy software systems. Companies often release improved versions of the core program due to the constant demand and growing competition in the software field. A variety of growth models have been created to track and measure reliability by software managers and engineers. In the testing phase, the fault content and size of the software system increase, and consequently, the fault content found and eliminated during each debugging process decreases in comparison to the fault content present at the initial stage. In this scenario, we can consider the software fault detection process to be stochastic. The fault detection process is expressed in terms of testing coverage with random effects. In this study, we construct a testing coverage-based software model incorporating random effect with change point. We have used different testing coverage functions such as Exponential and Delayed S-shaped to study the effect of randomness. Further, multi-release planning for the proposed model has been studied and validated on the real-time failure dataset from Tandem Computers with four releases. Different performance measures and goodness-of-fit have been presented using a graphical representation.

Keywords

Software reliability growth model testing coverage change point stochastic differential equation multi-release model

1. Introduction

The rapid growth in the field of software technology has significantly improved the quality of life and enhanced the development process of society. Software-based systems have become such an important element in modern day-to-day life, it is nearly difficult to carry out most of our daily activities without their assistance. As we have increased our dependence on software systems, it becomes important that they function reliably. The growing convergence of internet technology and electronic commerce has significantly increased the need for software products. As a result, developers face difficulty in fulfilling and impressing their clients. Software developers always try to look out for new techniques and pathways for clients’ happiness and market visibility. Software upgrades are one viable technique to achieve this aim. To build authentic and precise software with a smaller number of errors or faults is an ultimate take of a software developer which consumes time and effort. Technological advancements are occurring at a rapid pace, and these new developments, which are simply expansions of the original product, frequently act as if they are new products in the market. Generally, a new version is released as the software attains an operational reliability level as per the requirements of the firm. An upgrade refers to the replacement of the old version of the software with the new one. It is a difficult task to upgrade a software program. There may be a difference in terms of performance and functionality between the upgraded and current systems. In an upgrade full system is not changed, only specific components of the program are changed. The main aim of the developers is to enhance the software product, but there is also potential that the new version would deteriorate making the upgrading process complex and risky. To survive in the competitive market, software companies try to release new versions of the software because of a variety of reasons which are briefly provided in Fig. 1. The process of upgrading leads to the simultaneous growth in fault content. As a result, the testing crew of the software is always keen on learning about the problems existing in the program and, as a result, they constantly monitor the product’s efficiency. Safe system upgrades can enhance system behavior and help a company retain market share; nevertheless, unsafe system upgrades might result in serious errors in the system. At times, the new version performs as it was intended but the user may prefer the previous one [1]. The aforementioned problem is typically caused by software failure caused by mistakes, ambiguities, or misinterpretation of the criteria that the software is meant to meet unreliable code, insufficient testing before release, erroneous or unanticipated software usage, or other undiscovered factors. As there is a change in the selected component, not the entire system, it may lead to an escalation in the amount of fault content. The testers focus on learning about the defects in the program that will determine the functionality of upgraded software. In the history of this subject, many software reliability models have been proposed [2, 3, 4, 5, 6, 7].

Figure 1.

Need for the requirement of different releases of software.

Several types of research have been conducted to establish software reliability growth models (SRGM). The fault detection rate (FDR) is one of the most critical aspects that affect the reliability growth model of the software. The FDR assists in determining the efficiency of detecting faults using testing procedures and periods. FDR is assumed to be constant or monotonically rising by various researchers. Goel and Okumoto made the first attempt to propose Non-Homogeneous Poisson Process based SRGM [3]. Later, various SRGMs were proposed describing exponential and S-shaped curves by Yamada et al. [8], Obha and Osaki [8], Kapur and Garg [9], Pham [10], and Kapur et al. [11]. Moreover, the fault removal rate is heavily dependent on the experience of the testing team, fault density, size of the program, and testing effort. In real life, we cannot assume that these factors remain stable while the testing and removal process. So, it is right to consider that FDR may fluctuate due to fluctuations in the testing environment. The time at which FDR fluctuates or changes is considered to be a change point. So, when there is an increase or drop in the failure intensity function, we can say that the change point occurred. The first attempt to work in this area was made by Zhao [12]. Later researchers made their contribution by merging the concept of imperfect debugging and testing efforts to attain a certain level of desired reliability [13, 14, 15]. Kapur et al. proposed various SRGMs based on single and multiple change points [15, 16, 17, 18, 19]. Recently, in 2022 a changepoint-based SRGM was proposed incorporating a time-dependent testing effort [20]. Nidhi Nijhawan used time-dependent exponentiated Weibull-based FRF to analyze the effect of change in the environment on the growth model [21]. We have also considered the concept of change points in the testing periods in our proposed model.

Furthermore, it has been suggested that software reliability can be enhanced further by taking into account the impact of specific real-world challenges that occur throughout the testing process. One of the most significant considerations is testing coverage (TC) for the developers and as well as users [22, 23, 24, 25, 26, 27, 28]. The software developers can assess the nature and quality of the software by considering testing coverage and later they can regulate how much further resolution or effort is needed to improve the reliability. During the process of testing, the behavior of coverage growth is stated in terms of a function $c(t)$ which is percentage of code covered up to testing time t and is also known as the coverage growth function. Coverage in the Software reliability growth model helps to improve dependability and forecast failures more realistically and accurately. During the whole process of testing, the performance and accuracy are indicated using TC. Many types of research have been conducted that incorporate various time-dependent testing coverage functions such as S-shaped [29], Logarithmic- Exponential [30], Rayleigh [31, 32], Weibull [25], LogNormal [33]. A model was suggested by Malaiya et al. to explain the connection between reliability and TC. In their study, they examined the relationship between TC and defect coverage after developing a hypothesis based on the identification of the enumerable coverage metrics [34]. Later, Malaiya et al. provided a model that included TC, reliability along with testing duration and applied the model to four datasets. The author defined the TC and effort using the Log-arithmetic exponential model [24]. In 2003, Pham and Zhang suggested a cost-based SRGM incorporating TC and they evaluated the model’s goodness of fit using different datasets [29]. The suggested model was also compared to the traditional SRGMs, which led to a superior forecast. In 2014, Pham developed two models that take into account the TC’s ability to capture the unpredictability of the operational environment using the FDR as Log-log distribution [35]. SRGM was examined by Li and Pham in 2017, and they looked at TC as the rate of fault discovery in the context of an unpredictable operating environment [36].

The effect of randomness plays a vital role that impacts the rate of failure, the efficiency of eliminating fault, TC, change point, and so on. Kapur et al. proposed a model that handles the randomness by using the concept of Itô type of Stochastic Differential equation (SDE) [37, 38, 39, 40, 41]. The motivation behind our study is that none of the SRGMs discussed in the literature have focused on the testing coverage with the effect of randomness. In our study, we have studied the effect of randomness on testing coverage-based FDR using SDE with the change point concept. Our study considers TC function as Exponential and S-Shaped distribution functions. Firstly, the reason behind considering the exponential TC function is that FDR is constant throughout the testing process which handles the case in which the code is uniform. When the faults are observed they are immediately isolated [3]. Also, during the testing procedure, the failure intensity rate decreases which indicates that there is an enhancement in the quality of the code of the software. But, in a real-life situation, it increases first and decreases over time. Secondly, the reason behind considering the S-shaped TC function is to recognize the realness of the time delay between the detection/observation and removal of faults from software code. There are two phases of the testing procedure: fault detection and isolation [8].

The concept of multi-release has been widely studied by different researchers. Kapur et al. stated that the upgradation of the software version can generate new faults [1, 42, 43]. In 2019, Anu Gupta Aggarwal et al. suggested a model with a time-dependent FRF along with imperfect debugging. They also proposed a multi-release model of the software [4]. Later in 2020, V Verma et al. modeled FRF by studying the delayed S-shaped model with constant error generation [44]. Later, a multi-release model was generated for an open-source project based on FRF and TC by Tandon and Aggarwal [45]. In 2022, a release planning problem was explained by Anu Aggarwal and Jaiswal in which weights of the modules were calculated by using the Fuzzy Analytical Hierarchy process. They modeled two SRGM based on FRF and testing coverage [46]. A modified Weibull distribution was used to address FDR to create the reliability model along with the multi-release planning problem by Pradhan, Kumar and Dhar in 2022 [47]. It has been observed that most of the proposed SRGMs used easily predictable preset parameters. Moreover, there is instability and randomness in the operating field environment when software is released. In 2023, two multi-upgrade SRGMs were created by considering the random field operating environment. These SRGMs captured the uncertainty of FDR [48]. In our research, a testing coverage-based SRGMs has been proposed incorporating random effect and change points. The randomness is caused by some irregular fluctuation while covering software code during the testing process. Later, we incorporated a multi-release model into our study.

This research contains three stages: at stage one the general model has been developed using assumptions, at stage two the multi-release model has been developed and at stage three the models have been validated on a real-time Tandem Computer dataset. We have divided the paper into six sections as follows: The notations and the assumption used in our proposed model have been provided in section 2. The model development is well explained in section 3 in which the concept of irregular fluctuations and change points have been implemented. In section 4, multi-release modeling has been suggested for up to four releases. The numerical illustration has been provided in section 5 in which we have used four release datasets of Tandem Computers. Later, the results of parameter estimation and goodness-of-fit curves are presented. The conclusions related to the model in this paper have been drawn in section 6 with the future scope.

2. Notations

$N(t)$ : a random variable representing the number of faults detected at testing time $t$ .

$m(t)$ : Mean value function providing the expected number of faults detected or removed by time $t$ .

$a$ : Overall anticipated faults in the system before testing.

$\tau$ : Change-point.

$c(t)$ : Represents the portion of potential faults in the code identified up to time $t$ .

${c}^{\prime}(t)$ : Coverage rate.

$b(t)$ : Fault detection/removal rate.

$\sigma$ : A parameter representing the magnitude of randomness caused by irregular fluctuation.

$\gamma(t)$ : Standard Gaussian White Noise.

$F(t)$ : Cumulative Failure function before the change point.

$G(t)$ : Cumulative Failure function after the change point.

2.1 Basic assumptions

1.
The FDR of the software is modeled as a stochastic process.
2.
The FDR might change at any point in time called change-point.
3.
The software is subjected to failures caused by the remaining number of faults during execution.
4.
Faults are finite in number in the software code.
5.
The FDR rate can be represented in terms of testing coverage as $\frac{c^{\prime}(t)}{1-c(t)}$ .
6.
Testing Coverage is subjected to some randomness.
7.
All faults in the software are mutually independent.
8.
The debugging process is assumed to be perfect and during the process, no new faults were introduced.

3. Model development

The differential equation explaining the fault detection process is given as:

$\displaystyle\frac{dN(t)}{dt}=\begin{cases}\frac{c^{\prime}_{1}(t)}{1-c_{1}(t)% }[a-N(t)];&0\leqslant t\leqslant\tau\\ \frac{c^{\prime}_{2}(t)}{1-c_{2}(t)}[a-N(\tau)-N(t)];&t>\tau\end{cases}$ (1)

where, $\frac{c^{\prime}(t)}{1-c(t)}$ is FDR in terms of testing coverage and is a non-negative function. We also assume that the testing coverage is affected by some random aspects which are uncertain and affect the testing process. Therefore, we capture the stochastic property of the testing coverage by extending Eq. (1) to a realistic equation. We assume the irregular fluctuation in fault detection rate as: $\frac{c^{\prime}(t)}{1-c(t)}+\sigma\gamma(t)$ and extend the basic differential equation to a stochastic equation. So, the stochastic differential equation (SDE) is given as:

$\displaystyle\frac{dN(t)}{dt}=\begin{cases}\left[\frac{c^{\prime}_{1}(t)}{1-c_% {1}(t)}+\sigma\gamma(t)\right][a-N(t)];&0\leqslant t\leqslant\tau\\ \left[\frac{c^{\prime}_{2}(t)}{1-c_{2}(t)}+\sigma\gamma(t)\right][a-N(\tau)-N(% t)];&t>\tau\end{cases}$ (2)

We extend Eq. (2) to the SDE of an Itô type [37, 49].

$\displaystyle dN(t)=\begin{cases}\left(\frac{c^{\prime}_{1}(t)}{1-c_{1}(t)}-% \frac{1}{2}\sigma^{2}\right)[a-N(t)]dt+\sigma[a-N(t)]dw(t);&0\leqslant t% \leqslant\tau\\ \left(\frac{c^{\prime}_{2}(t)}{1-c_{2}(t)}-\frac{1}{2}\sigma^{2}\right)[a-N(% \tau)-N(t)]dt+\sigma[a-N(\tau)-N(t)]dw(t);&t>\tau\end{cases}$ (3)

where $W(t)$ represents the one-dimensional Wiener process and its time derivative is given by $\frac{dW(t)}{dt}=\gamma(t)$ . The Weiner process is a Gaussian process with the given properties [40]:

$\displaystyle\textit{Prob}[w(0)=0]=1,$ $\displaystyle E[w(t)]=0,$ $\displaystyle E[w(t)w(t^{\prime})]=\min[t,t^{\prime}],$

Now, we incorporate two types of testing coverage functions:

3.1 SRGM 1 – exponential testing coverage

The Fault detection rate for the exponential testing coverage model is:

$\displaystyle b(t)=\frac{c^{\prime}(t)}{1-c(t)}=\begin{cases}b_{1};&0\leqslant t% \leqslant\tau\\ b_{2};&t>\tau\end{cases}$ (4)

Now, the stochastic differential equation of the model provided in Eq. (3) using Eq. (4) can be written as:

$\displaystyle dN(t)=\begin{cases}\left(b_{1}-\frac{1}{2}\sigma^{2}\right)(a-N(% t))dt+\sigma(a-N(t))dw(t);&0\leqslant t\leqslant\tau\\ \left(b_{2}-\frac{1}{2}\sigma^{2}\right)(a-N(\tau)-N(t))dt+\sigma(a-N(\tau)-N(% t))dw(t);&t>\tau\end{cases}$ (5)

The Itô formula solution is used to solve Eq. (5) by considering at $t=0$ , $N(t)=0$ , and $t=\tau$ , $N(t)=N(\tau)$ , we get:

$\displaystyle N(t)=\begin{cases}a\left(1-e^{(-b_{1}t-\sigma W(t))}\right);&0% \leqslant t\leqslant\tau\\ a\left(1-e^{(-b_{1}\tau-b_{2}(t-\tau)-\sigma W(t))}\right);&t>\tau\end{cases}$ (6)

The expectation of $N(t)$ provides the MVF and is given by:

$\displaystyle m_{\textit{srgm1}}=\begin{cases}a\left(1-e^{\left(-b_{1}t+\frac{% \sigma^{2}t}{2}\right)}\right);&0\leqslant t\leqslant\tau\\ a\left(1-e^{\left(-b_{1}\tau-b_{2}(t-\tau)+\frac{\sigma^{2}t}{2}\right)}\right% );&t>\tau\end{cases}$ (7)

where $E(e^{-\sigma W(t)})=e^{\sigma^{2}t/2}$ , or:

$\displaystyle m_{\textit{srgm1}}=\begin{cases}aF(t);&0\leqslant t\leqslant\tau% \\ aG(t);&t>\tau\end{cases}$ (8)

3.2 SRGM 2 – delayed S-shaped testing coverage

The FDR for the delayed S-shaped testing coverage model is:

$\displaystyle b(t)=\frac{c^{\prime}(t)}{1-c(t)}=\begin{cases}\frac{b_{1}^{2}}{% 1+b_{1}t};&0\leqslant t\leqslant\tau\\ \frac{b_{2}^{2}}{1+b_{2}t};&t>\tau\end{cases}$ (9)

Now, the SDE of the model provided in Eq. (4) using Eq. (9) can be written as:

$\displaystyle dN(t)=\begin{cases}\left(\frac{b_{1}^{2}}{1+b_{1}t}-\frac{1}{2}% \sigma^{2}\right)(a-N(t))dt+\sigma(a-N(t))dw(t);&0\leqslant t\leqslant\tau\\ \left(\frac{b_{2}^{2}}{1+b_{2}t}-\frac{1}{2}\sigma^{2}\right)(a-N(\tau)-N(t))% dt+\sigma(a-N(\tau)-N(t))dw(t);&t>\tau\end{cases}$ (10)

The Itô formula solution is used to solve Eq. (10) by considering at $t=0$ , $N(t)=0$ , and $t=\tau$ , $N(t)=N(\tau)$ , we get:

$\displaystyle N(t)=\begin{cases}a\left(1-(1+b_{1}t)e^{(-b_{1}t-\sigma W(t))}% \right);&0\leqslant t\leqslant\tau\\ a\left(1-\frac{(1+b_{1}\tau)}{(1+b_{2}\tau)}(1+b_{2}t)e^{(-b_{1}\tau-b_{2}(t-% \tau)-\sigma W(t))}\right);&t>\tau\end{cases}$ (11)

The expectation of $N(t)$ provides the MVF and is given by:

$\displaystyle E(N(t))=m(t)=\begin{cases}a\left(1-(1+b_{1}t)e^{\left(-b_{1}t+% \frac{\sigma^{2}t}{2}\right)}\right);&0\leqslant t\leqslant\tau\\ a\left(1-\frac{(1+b_{1}\tau)}{(1+b_{2}\tau)}(1+b_{2}t)e^{\left(-b_{1}\tau-b_{2% }(t-\tau)+\frac{\sigma^{2}t}{2}\right)}\right);&t>\tau\end{cases}$ (12)

or:

$\displaystyle m_{\textit{srgm2}}=\begin{cases}aF(t);&0\leqslant t\leqslant\tau% \\ aG(t);&t>\tau\end{cases}$ (13)

4. Multi-release model

Figure 2.

Multi-release model.

The multi-release model has been suggested incorporating perfect debugging, change point, and testing coverage with random effect. The multi-release model with change point and release time has been shown in Fig. 2. Using Eqs (10) and (15), the MVFs for different releases can be obtained. The notations used in the multi-release model are:

$a_{i}$ : Initial Fault content for $i^{\text{th}}$ version; $i=$ 1,2,3,4.

$t_{i}$ : Time of release for $i^{\text{th}}$ version.

$\tau_{i}$ : change point in $i^{\text{th}}$ version.

$F_{ik}(t)$ : cumulative failure function before the change point for $i^{\text{th}}$ version corresponding to $j^{\text{th}}$ model.

$G_{ik}(t)$ : cumulative failure function after the change point for $i^{\text{th}}$ version corresponding to $j^{\text{th}}$ model.

$m_{ik}(t)$ : MVF for the $i^{\text{th}}$ version corresponding to the $k^{\text{th}}$ model.

4.1 Release 1

It is the initial stage in developing the product’s reputation across users. Teams of testers work rigorously to eliminate as many faults as possible throughout testing. Consider that the software is initially released at time $t_{1}$ and the MVF for the fault detection process is given as:

$\displaystyle m_{1k}=\begin{cases}a_{1k}F_{1k}(t);&0\leqslant t\leqslant\tau_{% 1}\\ a_{1k}G_{1k}(t);&\tau_{1}<t\leqslant t_{1}\end{cases}$ (14)

4.2 Release 2

To survive in today’s fierce world of competition, software companies must keep themselves up-to-date on the newest needs and innovations. To lure new consumers, they constantly add improvements to the software. The newly added improvements or features necessitate changes to the source code, which results in the creation of new faults. The team of testers addresses faults discovered during the operational period of the previous version, as well as faults discovered as a result of the newly introduced features in the recently released version. Now, at time $t_{2}$ the software is released with a new version in the code for the second time, it contains newly detected faults and a few faults from the previous release.

$\displaystyle m_{1k}=\begin{cases}[a_{2k}+a_{1k}(1-F_{1k}(t))]F_{2k}(t-t_{1});% &t_{1}\leqslant t\leqslant\tau_{2}\\ G_{2k}(t-t_{1});&\tau_{2}<t\leqslant t_{2}\end{cases}$ (15)

4.3 Release 3

Similarly, it has been considered that the prior release has an impact on the MVF of the third release. In the third release at time $t_{3}$ , the software contains newly detected faults from release 3 and some additional faults which could not be taken out from the second release [50].

$\displaystyle m_{3k}(t)=\begin{cases}[a_{3k}+a_{2k}(1-F_{2k}(t_{2}))+a_{1k}(1-% F_{1k}(t_{1}))(1-F_{2k}(t_{2}))]\\ \quad F_{3k}(t-t_{2});&t_{2}\leqslant t\leqslant\tau_{3}\\ \\ \quad G_{3k}(t-t_{2});&\tau_{3}<t\leqslant t_{3}\end{cases}$ (16)

4.4 Release 4

Similarly, the fourth release at time $t_{4}$ contains faults from the fourth release and a few faults from the previous releases.

$\displaystyle m_{4k}(t)=\begin{cases}[a_{4k}+a_{3k}(1-F_{3k}(t_{3}))+a_{2k}(1-% F_{2k}(t_{2}))(1-F_{3k}(t_{3}))+\\ \quad a_{1k}(1-F_{1k}(t_{1}))(1-F_{2k}(t_{2}))(1-F_{3k}(t_{3}))]F_{4k}(t-t_{3}% );&t_{3}\leqslant t\leqslant\tau_{4}\\ G_{4k}(t-t_{3});&t_{3}\leqslant t\leqslant\tau_{4}\\ \end{cases}$ (17)

5. Numerical illustration and results

5.1 Model validation

After the formulation of the multi-release SRGM, we validate the model using Non-linear Least square estimation technique (NLLS) in SPSS on the dataset. The parameters of the models are the first estimate using NLLS. Then, the number of cumulative faults is estimated. Later, the goodness-of-fit measures are explained as follows:

i)
Coefficient of determination ( $R^{2}$ ):

It measures the amount of variation in the dependent variable. It is in the range of 0 to 1 and if the value is closer to 1 then it implies a better fit.

$\displaystyle R^{2}=1-\frac{\sum_{i=1}^{k}(m_{i}-\widehat{m}(t_{i}))^{2}}{\sum% _{i=1}^{k}(m_{i}-\sum_{j=1}^{k}m_{j}/k)^{2}}$
ii)
Mean Square Error:

Table 1
Results of parameter estimation

Releases Models $a$ $b_{1}$ $b_{2}$ $\sigma$

1 SRGM 1 103.653 0.83 0.81 0.0008617

SRGM 2 96.769 0.081 0.219 0.0000201

2 SRGM 1 123.721 0.72 0.291 0.0051

SRGM 2 118.341 0.261 0.246 0.00017

3 SRGM 1 65.875 0.065 0.152 0.014

SRGM 2 58.720 0.273 0.729 0.003340

4 SRGM 1 38.912 0.61 0.256 0.00223

SRGM 2 37.856 0.195 0.366 0.0000812

Table 2
Goodness-of-fit measures values for proposed models

Releases Models $R^{2}$ Bias MSE MAE Variance RMPSE

1 SRGM 1 0.986 0.065 6.978755 2.3495 2.71301 2.713853

SRGM 2 0.988 0.747 6.9913 2.203 3.020618 3.1116530

2 SRGM 1 0.986 $-$ 0.06157 7.744026 2.498 2.861164 2.86182

SRGM 2 0.995 $-$ 0.37736 9.488131 2.41305 3.235152 3.327086

3 SRGM 1 0.987 $-$ 0.561 4.52031 1.8633 2.4420 2.505821

SRGM 2 0.996 0.44083 3.53189 1.4025 2.11873 2.16409

4 SRGM 1 0.988 $-$ 0.1689 1.478789 1.08526 1.2814 1.291360

SRGM 2 0.995 0.32256 1.302109 0.957894 1.383543 1.420543

Figure 3.
Release 1: Goodness-of-fit curve.

Figure 4.
Release 2: Goodness-of-fit curve.

Figure 5.
Release 3: Goodness-of-fit curve.

Figure 6.
Release 4: Goodness-of-fit curve.

It is the deviation between actual and estimated values. If the value is less then it implies a better fit.

$\displaystyle\textit{MSE}=\frac{\sum_{i=1}^{k}(m_{i}-\widehat{m}(t_{i}))^{2}}{k}$
iii)
Mean Absolute Error:

It is similar to MSE but it measures deviation with absolute values. If the value is less then it implies a better fit.

$\displaystyle\textit{MAE}=\frac{\sum_{i=1}^{k}|(m_{i}-\widehat{m}(t_{i}))|}{k-p}$
iv)
Bias

The average deviation between the actual and estimated values gives the bias. If the value is less then it implies a better fit.

$\displaystyle\textit{Bias}=\frac{\sum_{i=1}^{k}|(\widehat{m}(t_{i})-m_{i})|}{k}$
v)
Variance

It is given by the standard deviation of the bias.

$\displaystyle\textit{Variance}=\sqrt{\frac{\sum_{i=1}^{k}(m_{i}-\widehat{m}(t_% {i})-\textit{Bias})^{2}}{k-1}}$
vi)
Root Mean Square Percentage Error

It measures how well the estimation has been done. It is the standard deviation of the residuals. If the value is less then it implies a better fit.

$\displaystyle\textit{RMSPE}=\sqrt{\textit{Variance}^{2}+\textit{Bias}^{2}}$

5.2 Data analysis

Releases	Models	$a$	$b_{1}$	$b_{2}$	$\sigma$
1	SRGM 1	103.653	0.83	0.81	0.0008617
	SRGM 2	96.769	0.081	0.219	0.0000201
2	SRGM 1	123.721	0.72	0.291	0.0051
	SRGM 2	118.341	0.261	0.246	0.00017
3	SRGM 1	65.875	0.065	0.152	0.014
	SRGM 2	58.720	0.273	0.729	0.003340
4	SRGM 1	38.912	0.61	0.256	0.00223
	SRGM 2	37.856	0.195	0.366	0.0000812

Releases	Models	$R^{2}$	Bias	MSE	MAE	Variance	RMPSE
1	SRGM 1	0.986	0.065	6.978755	2.3495	2.71301	2.713853
	SRGM 2	0.988	0.747	6.9913	2.203	3.020618	3.1116530
2	SRGM 1	0.986	$-$ 0.06157	7.744026	2.498	2.861164	2.86182
	SRGM 2	0.995	$-$ 0.37736	9.488131	2.41305	3.235152	3.327086
3	SRGM 1	0.987	$-$ 0.561	4.52031	1.8633	2.4420	2.505821
	SRGM 2	0.996	0.44083	3.53189	1.4025	2.11873	2.16409
4	SRGM 1	0.988	$-$ 0.1689	1.478789	1.08526	1.2814	1.291360
	SRGM 2	0.995	0.32256	1.302109	0.957894	1.383543	1.420543

The validation and estimation of the parameters of our model are done using by gathering real-time software data from Tandem Computers for four releases [51]. The data collection contains failure information from four significant software product releases. The actual data curve was drawn for all the releases to pinpoint the kinks which are nothing but change-point. It was observed that the changed point was observed at the 8 ${}^{\text{th}}$ , 26 ${}^{\text{th}}$ , 42 ${}^{nd}$ , and 56 ${}^{\text{th}}$ week of the four releases respectively. The estimated parameters for both SRGM 1 and 2 are provided in Table 1. For SRGM 1, the value of $b_{1}$ and $b_{2}$ is lowest in Release 3 in comparison to other releases. It may be because the dataset is exponential while $b_{1}$ and $b_{2}$ are highest for Release 1 which implies a high S-shape of the dataset from Release 1. In the case of SRGM 2, the value of $b_{1}$ and $b_{2}$ is least for Release 1 in comparison to other releases while $b_{1}$ and $b_{2}$ are high for Release 3 which implies a high S-shape of the dataset from Release 3.

The goodness-of-fit analysis for both the SRGMs corresponding to each release is provided in Table 2. In the case of SRGM 1, the highest $R^{2}$ is observed for release 4, and the lowest MSE for release 4, so we can interpret that the dataset is exponential. In the case of SRGM 2, the maximum $R^{2}$ is observed for Release 3 and the lowest MSE for Release 4, so in this case we can say the model is S-shaped and the dataset is exponential.

From Table 2, it can be commemorated that the $R^{2}$ value is high for releases 3, and 4 for SRGM 1 and is high for releases 2, 3, and 4 for SRGM 2. It can be said that our models fit the data significantly well. When the proposed models are compared to SRGM without a change point from Kapur et al. (2012), it is observed that our proposed model is distinctive [41]. Our proposed models perfectly capture the idea of the change point in testing coverage-based SRGM with irregular fluctuations. The goodness-of-fit curves are presented in Figs 3–6 for each release of both models.

6. Conclusion, limitations, and future scope

For most company organizations, handling the sales of technical products along with reputation in a cutthroat market or technological developments is a serious issue, particularly for high-tech products like software. Because of the significant costs and dangers involved in developing new software, as well as the intense market rivalry, businesses are compelled to develop follow-up or new versions to survive. The improved versions are intended to enhance the market’s potential adopters, but if the improved version is disliked by the market, it might result in significant financial and reputational loss. The assessment of the improved software system after release is a difficult process since it must be done to ensure that the many releases/multiple versions of the software are helpful for the developer. Because incorporating any new capability into an already complicated system increases complexity. In this research, we offer a testing coverage SRGM with randomness and change points along with a multi-release model. No software code is completely bug-free, as it is typically observed in practice. Testing of software is therefore a continual activity. We argue that there is a potential that certain flaws in the previously released version of software may be discovered and fixed when testing a new code. This aids in the removal of increasing software faults and results in highly dependable software. the two types of Testing Coverage functions have been incorporated: Exponential and delayed S-shaped functions. The advantage of using exponential TC is that the FDR is constant throughout the testing process which handles the case in which the code is uniform and as soon as the faults are observed they are immediately isolated. Also, during the testing procedure, the failure intensity rate decreases which indicates that there is an enhancement in the quality of the code of the software. But, in a real-life situation, it increases first and decreases over time. Secondly, the reason behind considering the S-shaped TC function is to recognize the realness of the time delay between the detection/observation and removal of faults from software code. There are two phases of the testing procedure: fault detection and isolation

The four-release dataset is used to estimate the suggested multi-release model based on Testing coverage SRGM with change point. The suggested model yields significant parameter estimations and goodness-of-fit measures. The results of the parameters are compared to the unified SDE proposed in 2012 without a change point [41]. The findings reveal that our suggested model provides a superior fit. In the future, we will attempt to develop more models in the same field using other traditional testing coverage functions. By enabling multiple change points, and imperfect fault debugging, the suggested model may be expanded using the change-point concept. The concept of different fault severity can also be incorporated in the future.

Footnotes

Acknowledgments

We would like to express our sincere gratitude to all the individuals and organizations that have contributed to this research paper.

First and foremost, we are grateful to the Department of Operational Research at the University of Delhi for providing us with the resources and support. I would also like to thank our research fellows at the Institution for their feedback and support throughout the research process. Their contributions have been invaluable in helping us to understand the topic and draw meaningful conclusions

Secondly, we would also like to show our gratitude to the editorial board members of the journal, and we also thank the reviewers who will be providing their so insights.

References

Kapur

Tandon

Kaur

. Multi up-gradation software reliability model. In: 2010 2nd International Conference on Reliability, Safety and Hazard-Risk-Based Technologies and Physics-of-Failure Methods (ICRESH). IEEE; 2020.

Bittanti

. Software reliability modelling and identification. Vol. 341. Springer Science & Business Media; 1998.

Goel

Okumoto

, Time-dependent error-detection rate model for software reliability and other performance measures. IEEE transactions on Reliability. 1979; 28(3): 206-211.

Aggarwal

, et al. Multi-release software reliability growth assessment: an approach incorporating fault reduction factor and imperfect debugging. International Journal of Mathematics in Operational Research. 2019; 15(4): 446-463.

Almering

, et al. Using software reliability growth models in practice IEEE software. 2007; 24(6): 82-88.

Aggarwal

Kapur

Nijhawan

. A discrete SRGM for multi-release software system with faults of different severity. International Journal of Operational Research. 2018; 32(2): 156-168.

Anand

Verma

Aggarwal

. 2-Dimensional multi-release software reliability modelling considering fault reduction factor under imperfect debugging. Ingeniería Solidaria. 2018; 14(25): 1-12.

Yamada

Ohba

Osaki

. S-shaped software reliability growth models and their applications. IEEE Transactions on Reliability. 1984; 33(4): 289-292.

Kapur

Garg

. A software reliability growth model for an error-removal phenomenon. Software Engineering Journal. 1992; 7(4): 291-294.

10.

Pham

. Software reliability. Springer Science & Business Media; 2000.

11.

Kapur

, et al. Software reliability assessment with OR applications. 2011.

12.

Zhao

. Change-point problems in software and hardware reliability. Communications in Statistics-Theory and Methods. 1993; 22(3): 757-768.

13.

Chang

. Estimation of parameters for nonhomogeneous Poisson process: Software reliability with change-point model. Communications in Statistics-Simulation and Computation. 2001; 30(3): 623-635.

14.

Shyur

. A stochastic software reliability model with imperfect-debugging and change-point. Journal of Systems and Software. 2003; 66(2): 135-141.

15.

Kapur

, et al. Testing effort control using flexible software reliability growth model with change point. International Journal of Performability Engineering. 2006; 2(3): 245.

16.

Kapur

, et al. Software reliability growth model with change-point and effort control using a power function of the testing time. International Journal of Production Research. 2008; 46(3): 771-787.

17.

Kapur

, et al. Software reliability growth modelling for errors of different severity using change point. International Journal of Reliability, Quality and Safety Engineering. 2007; 14(4): 311-326.

18.

Goswami

Khatri

Kapur

. Discrete software reliability growth modeling for errors of different severity incorporating change-point concept. International Journal of Automation and Computing. 2007; 4: 396-405.

19.

Kapur

, et al. General framework for change point problem in software reliability and related release time problem. International Journal of Reliability, Quality and Safety Engineering. 2009; 16(6): 567-579.

20.

Pradhan

Dhar

Kumar

. Testing-Effort based NHPP Software Reliability Growth Model with Change-point Approach. Journal of Information Science & Engineering. 2022; 38(2).

21.

Dhaka

Nijhawan

. Effect of change in environment on reliability growth modeling integrating fault reduction factor and change point: a general approach. Annals of Operations Research. 2022; 1-35.

22.

Huang

Kuo

Lyu

. An assessment of testing-effort dependent software reliability growth models. IEEE Transactions on Reliability. 2007; 56(2): 198-211.

23.

Cai

Lyu

. Software reliability modeling with test coverage: Experimentation and measurement with a fault-tolerant software project. In: The 18th IEEE International Symposium on Software Reliability (ISSRE’07). IEEE; 2007.

24.

Malaiya

, et al. Software reliability growth with test coverage. IEEE Transactions on Reliability. 2002; 51(4): 420-426.

25.

Gokhale

, et al. Unification of finite failure non-homogeneous Poisson process models through test coverage. In: Proceedings of ISSRE’96: 7th International Symposium on Software Reliability Engineering. IEEE; 1996.

26.

Chang

, et al. A testing-coverage software reliability model with the uncertainty of operating environments. International Journal of Systems Science: Operations & Logistics. 2014; 1(4): 220-227.

27.

Chatterjee

Singh

. A NHPP based software reliability model and optimal release policy with logistic – exponential test coverage under imperfect debugging. International Journal of System Assurance Engineering and Management. 2014; 5(3): 399-406.

28.

Chatterjee

Shukla

. Effect of Test Coverage and Change Point on Software Reliability Growth Based on Time Variable Fault Detection Probability. J Softw. 2016; 11(1): 110-117.

29.

Pham,

Zhang

. NHPP software reliability and cost models with testing coverage. European Journal of Operational Research. 2003; 145(2): 443-454.

30.

Huang

Lin

. Software reliability analysis by considering fault dependency and debugging time lag. IEEE Transactions on Reliability. 2006; 55(3): 436-450.

31.

Xie

, et al. A study of the modeling and analysis of software fault-detection and fault-correction processes. Quality and Reliability Engineering International. 2007; 23(4): 459-470.

32.

Vouk

. Using reliability models during testing with non-operational profiles. In: Proceedings of the 2nd Bellcore/Purdue workshop on issues in Software Reliability Estimation. Citeseer; 1992.

33.

Park

Lee

Park

. A class of coverage growth functions and its practical application. Journal of the Korean Statistical Society. 2008; 37(3): 241-247.

34.

Malaiya

, et al. The relationship between test coverage and reliability. In: Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering. IEEE; 1994.

35.

Pham

. Loglog fault-detection rate and testing coverage software reliability models subject to random environments. Vietnam Journal of Computer Science. 2014; 1(1): 39-45.

36.

Pham

. NHPP software reliability model considering the uncertainty of operating environments with imperfect debugging and testing coverage. Applied Mathematical Modelling. 2017; 51: 68-85.

37.

Shigeru

Akio

. A stochastic differential equation model for software reliability assessment and its goodness-of-fit. International Journal of Reliability and Applications. 2003; 4(1): 1-12.

38.

Kapur

, et al. A generalised software growth model using stochastic differential equation. Communication in Dependability and Quality Management Belgrade, Serbia. 2007; 34.

39.

Tamura

Yamada

. A flexible stochastic differential equation model in distributed development environment. European Journal of Operational Research. 2006; 168(1): 143-152.

40.

Kapur

, et al. Stochastic differential equation-based flexible software reliability growth model. Mathematical Problems in Engineering. 2009; 2009.

41.

Kapur

, et al. A unified scheme for developing software reliability growth models using stochastic differential equations. International Journal of Operational Research. 2012; 15(1): 48-63.

42.

Kapur

Sachdeva

Singh

. Optimal cost: a criterion to release multiple versions of software. International Journal of System Assurance Engineering and Management. 2014; 5(2): 174-180.

43.

Kapur

, et al. Two dimensional multi-release software reliability modeling and optimal release planning. IEEE Transactions on Reliability. 2012; 61(3): 758-768.

44.

Verma

Anand

Aggarwal

. Reliability Assessment of Multi-release Software System Under Imperfect Fault Removal Phenomenon. In: Decision Analytics Applications in Industry. Springer; 2020. pp. 367-380.

45.

Tandon

Aggarwal

. Testing coverage based reliability modeling for multi-release open-source software incorporating fault reduction factor. Life Cycle Reliability and Safety Engineering. 2020; 9(4): 425-435.

46.

Aggarwal

Jaiswal

. Multi-objective Release Time Problem for Modular Software using Fuzzy Analytical Hierarchy Process. In: Optimization Models in Software Reliability. Springer; 2022. pp. 159-191.

47.

Pradhan

Kumar

Dhar

. Modeling Multi-Release Open Source Software Reliability Growth Process with Generalized Modified Weibull Distribution. Evolving Software Processes: Trends and Future Directions. 2022; 123-133.

48.

Mishra

Kapur

Aggarwal

. A generalized multi-upgradation SRGM considering uncertainty of random field operating environments. International Journal of System Assurance Engineering and Management 2023; 1-9.

49.

Øksendal

. Stochastic differential equations. Springer; 2003. pp. 65-84.

50.

Kapur

Aggarwal

Nijhawan

. A discrete SRGM for multi release software system. International Journal of Industrial and Systems Engineering. 2014; 16(2): 143-155.

51.

Wood

. Predicting software reliability. Computer. 1996; 29(11): 69-77.