Abstract
The operational environment of software differs from the debugging environment. Therefore, the study explores the impact of irregular consumption by diverse users on the development cost of software, reliability of software, and release decisions. To accomplish this, a release model has been formulated considering logistic testing coverage-based reliability growth model built as a stochastic process including the error generation. The stochastic nature has been captured by considering the noise factor due to irregular fluctuations occurring during testing while usage uncertainty has been captured by introducing a constant parameter in the cost function of the operational phase.
It is assumed that the testing phase cost is affected by the noise and the operation phase cost is affected by the severity of noise which is the result of uncertain usage by users. The model was evaluated against a real failure dataset. The release model creates a trade-off between software development cost, release timing, and reliability aspirations. This study contributes to software reliability literature and provides insights to practitioners to make software release decisions. The sensitivity analysis results give information about various aspects during the operational phase that affect the overall development cost.
Keywords
Introduction
In light of the revolution of emerging technologies, firms dealing with software development are feeling greater strain internationally to provide novel technologies that represent significant breakthroughs and give an edge over others. These evolving businesses face challenges to retain their current market share and draw new clients [1, 2]. The dependence of mankind on various fields like medicine, transportation, education, and tourism makes it important for software firms to develop products with higher reliability levels so that there is no failure during execution. Such failures can cause loss of property, and money and in critical cases, it can lead to loss of life. However, due to complicated code structures, it is not realistic to eradicate all defects during testing but it can be minimized. Errors can be introduced into the program’s code at any time throughout the development process [3].
Consumers choose long-lasting, high-performing products. Organizations prioritize developing reliable applications to prevent costly failures, safety concerns, and legal fines. When developing software, it is important to consider the frequency of failures, estimate testing effort, cease testing (testing stop time), and release timing [4, 5]. The continuous maintenance effort for a software application must be anticipated and estimated. This helps management plan and schedule activities accordingly. Some examples of this include how long the warranty should last once the software is released in the market, when the product should be released into the market, the number and severity of defects that can be expected, the number of software professionals (practitioners) required to support the product for development and maintenance [6].
The Software Reliability Growth Model (SRGM) is a mathematical instrument for Software Engineering that evaluates software, develops test plans and schedules, and monitors changes in reliability effectiveness [7]. Over the last two decades, many SRGMs have demonstrated that the connection between testing time and fault removal is either exponential, S-shaped or a combination of the two [7, 8, 9, 10, 11, 12]. Various sorts of errors in software need distinct methodologies and testing efforts to be eliminated [13, 14].
Several defects are identified and corrected to achieve the reliability goal for the software during the testing process and then it is released to the market. Customers uncover some defects during usage and report them to the developer, and then the team works on the existing code to remove the pain points and add some additional functionalities and then releases an improved version of the software. This version incorporates the solutions to the issues encountered by users while using the previous release and it also incorporates the latest technological developments that are in high demand. In this situation, the defects remaining in the system are viewed as a stochastic process [15].
Several researchers have worked on this concept and developed SRGMs using stochastic processes to incorporate irregular fluctuations. .Shigeru and Akio [16] proposed a basic SRGM to assess the detection and correction of faults leading to growth in the reliability of the software. The model used the Stochastic Differential Equation (SDE) of Itô type to derive the various standards related to using the stochastic process’s probability distribution. Later, they came up with a customizable SDE Model to describe the process of improving the reliability of software by detecting and removing the faults in an environment where different teams are involved in planning, analyzing, coding, and testing [17]. This type of environment with decentralized teams to work on various activities is known as the distributed development environment. Instead of the Non-Homogenous Poisson Process (NHPP), .Lee, and Kim [18] used SDEs to portray a per-fault detection process consisting of erratic variations, and they took into account the fault detection process dependent on testing duration
The work presented in this manuscript employs SDEs to depict the failure behavior of software demonstrating irregular fluctuations. This research reflects the implications of actual situations influencing the testing process, like tester’s knowledge and expertise affect the rate at which error is identified in the code after the failure occurs and then its removal from the software, the portion of code tested also influences the process of removing faults, the intricacy of code could generate additional errors in the software system during debugging process which eventually affects the rate at which error is identified in the code after the failure occurs and then its removal from the software and irregular fluctuations known as white noise affects the fault removal rate [19]. To tackle this, an SRGM that accounts for the amount of code covered during testing at time
Rapid technological advancements necessitate frequent software upgrades. These upgrades foster a long-term commercial connection between clients and developers. The developer releases the software in the market with time by upgrading the software in each release as per the requirements of the users. This research aims to reduce software development costs by considering both external and internal environmental elements throughout the product’s life cycle, including testing coverage. The operational environment of the software life cycle differs from its debugging environment because of the existence of uncertainty during its use by diverse users. We will investigate the impact of changes due to irregular fluctuations in software testing, the severity of the irregular fluctuation during the operational phase because of diverse users, and fault elimination costs on software release [22, 23].
To accomplish the aforementioned research goals, we will create a mathematical representation that depicts the failure process while taking into account the conditions listed below:
error generation phenomenon percentage of code covered during testing (during the process of identifying a fault in software during testing the part of the code is considered for testing instead of testing the complete code at once.) unusual variations present during testing
Further, the suggested mathematical model is utilized in an optimization model that is used for managing the entry of the software system into the market for users while aiming to reduce the overall development costs incurred during the process. This helps to determine the optimal testing periods for developers in the above-mentioned conditions [14, 24, 25, 26]. The optimization model consists of three cost components related to testing (software is in the hands of experts to identify and remove the faults) and operational phases (period after release of the product, here the software is in the hands of diverse types of users).
The conditions during the operating environment (when software has been released into the market and is extensively used by the users) are different from the debugging environment (the team of testers regressively tests the software to remove the maximum number of faults from the software). To address this difference, a constant parameter
The research paper consists of five sections. 1st section i.e. introduction section is going on. After that, we have discussed the past developments of SDE in software reliability literature. In Section 3 the SRGM and Optimization model is formulated for the software with multiple releases while considering the parameters affecting the fault removal process. In Section 4 the model developed in the section is fitted on the failure dataset of tandem computers. This dataset presents the faults observed in the computer with testing. This helps in validating the model for the software fault datasets. Further in this section, the influence of random changes in the severity of fluctuations is investigated. In the last section of the manuscript, the conclusions of the study are presented along with that the prospects of the concepts have also been discussed.
This section discusses the past developments related to the concepts utilized in the study. These studies assist in comprehending the necessity of analyzing current volatility and identifying research gaps.
Gokhale, Philip [36] developed a unified NHPP SRGM where the fault removal process was based on the testing coverage. Later, exponential, Weibull, and logistic testing coverage-based SRGMs were also proposed by them [37]. Pham and Zhang [20] proposed an SRGM that represents a failure curve in the form of
INOUE and YAMADA [38] gave a 1-D TC-based SRGM assuming that it is affected by the testing time and observed that coverage is highly impacted by the skills of the testers. Li and Pham [39] proposed TC-based SRGM by representing fault detection rate (FDR) as TC and validated the model on real-life data under the uncertainty in the operating environment and later Presented an SRGM using testing coverage and considered fault removal efficiency and error generation [39]. Aggarwal and Kumar [40] used three TCF were used to generate multi-release SRGM namely exponential, logistic, and S-shaped functions. Bibyan and Anand [41] proposed a unified SRGM using different testing coverage functions and estimated parameters using a neural network.
To capture the irregular fluctuation as noise, SDE-based SRGM gas has been used by several researchers in the past. The first attempt to use SDE was made by Yamada, and Kimura [42]. According to Shigeru and Akio [16], the faults that could not be removed from the program may be modelled as a stochastic process utilizing Itô-type SDE to generate SRGMs with different failure curves such as exponential failure curve, delayed S-shaped failure curve, and inflected S-shaped failure curve. Shyur [43] incorporated the parameters for the imperfect debugging environment and change in FDR in SDE-based SRGM. The change in FDR is also termed a change point.
Tamura and Yamada [17] Suggested flexible SDE-based SRGM for open-source software. The model discussed by them explains different patterns for reliability growth while considering the impact of different testing efforts employed to achieve reliability goals for the software. Kapur, Anand [44] proposed a generalized flexible SDE-based SRGM and extended the work by studying three types of faults namely simple, hand, and complex using SDE on Itô type [45]. The model discussed by them could explain the reliability behavior of software showing
Kapur, Anand [48] created a unified framework to develop SRGM using SDE. The authors proposed an SRGM that determines the number of faults removed from the considering that the process of removing faults is affected by the efforts employed during the testing process. Pham and Pham [49] incorporated the concept of environmental factors into their SDE-based SRGM. The SRGM model describes FDR using a Weibull distribution incorporating the influence of various environmental conditions (related to various phases of software development) quality of the software. Chatterjee and Chaudhuri [50] presented the use of SDE-based SRGM in the release planning model.
The suggested method determines release time by uniting metaheuristic algorithms and considering the budget affected by randomness under an intuitionistic fuzzy environment. Simulation findings show that the suggested strategy is successful at improving software dependability while minimizing costs. Later, they suggested an SDE-based SRGM in which fault detection is connected to the severity of fault incidence and fault correction is tied to the necessity of fault correction. The problem developed helps to create a tradeoff between the cost incurred in development and the reliability achieved after testing. The cost constraint consisted of coefficients given by interval type-2 fuzzy numbers. The type-2 fuzzy numbers are more accurate to reflect the variability of specifications given for software development [51]. Zheng, Yang [52] presented an SRGM incorporating fluctuation.
The SDE-based SRGM represented the disruption in the continuity due to uncertain random events. Bibyan and Anand [19] incorporated the concept of multi-release of the software by proposing SDE-based SRGM using different testing coverage functions. Later, the model was extended by incorporating an error generation parameter and the change point concept. A release model was suggested to capture the effect of irregular fluctuations on the cost used for software development and to observe its effect on the time decided for the release of the product [53, 54, 55, 56]. Table 1 summarizes SDE-based Models and our proposed work.
SDE literature
SDE literature
In this work, we will look at the impact of random fluctuations (white noise) due to uncertain events happening during the fault removal process on the ideal time to release the last release of multi-release software into the market. A logistic testing coverage-based SRGM is used, which accounts for the variations due to noise. It is modelled using SDE. The proposed SRGM also evaluates the influence of errors that are generated in the software when a particular fault is removed during testing on reliability achieved by the software. The existence of irregular fluctuations during testing, as well as their severity throughout the operating phase, affect the fixed cost parameter. Additionally, the sensitivity of the severity of fluctuation in this model has been investigated.
Here, under this section, we will develop an SRGM considering logistic testing coverage function incorporating irregular fluctuations during the testing phase and error generation phenomenon and further we will discuss a release model to determine the optimal testing duration that takes into account the impact of uncertain usage environment. The severity of the irregular fluctuations tends to change during the operational phase due to the diverse range of users.
SRGM development using SDE
Let’s start with a discussion of the notations and assumptions needed to create the SRGM.
Notations
Assumptions
The FDR is modeled using SDE. The FDR is affected by the leftover faults i.e. the faults that have not been removed from software till that time. The software may fail or cause some error in functioning during execution due to the faults already present in the software. The faults present in the software are mutually dependent i.e. action taken on a fault affects other faults also. The debugging process is not perfect i.e. some additional defects may get introduced while debugging The number of faults present in the software at a particular time changes due to the addition of errors during debugging and the removal of additional faults. The faults getting added are proportional to the amount of time spent testing it. The testing and operational phases differently affect the reliability of the software. The uneven variations (during the testing phase) and the uncertain usage of software (during the operational phase) have an impact on the program’s reliability. The FDR is expressed in terms of testing coverage and is mathematically given as
Using the abovementioned notations, the corresponding differential equation to depict the fault removal phenomenon by time
Equation (1) can also be represented in terms of testing coverage: (By using assumption (viii))
where,
It has been assumed in this study, that the error generation is dependent on testing time (assumption vi)
The standard Gaussian white noise has been considered to represent the irregular fluctuations and is denoted as
Hence, Eq. (3) can be written using Eq. (3) as
And,
Equation (6) is written as SDE of an Itô type and is given by Eq. (7):
where,
Probability Expectation Expectation
Now, the Itô solution has been applied to Eq. (7) utilizing conditions
The expectation of Eq. (8) has been taken:
Now considering the Logistic testing coverage function (assumptions ii. iii and iv)
On putting the Eq. (10) in Eq. (9), We have
The final Eq. (11) obtained after solving Eq. (10) using Eq. (9) presents the cumulative number of defects a tester was able to remove within time
For release 1 the MVF is
where
And
For release 2 the MVF is
where
For
where
Equation (14) presents
The SRGM developed in the previous section will be considered to find the best time to bring the
Testing cost is linearly related to the number of faults removed from the software. The cost of removing faults during operations is linearly related to the number of faults latent in the software. The irregular fluctuations are affecting testing costs by constant severity.
P1 defines the release problem in words:
The total cost of developing the
where the notations carry the following meanings:
The cost model developed in this sub-section shows that it consists of three cost components (fixed cost of testing, testing cost with time when faults are removed and operational cost when the software is in the hands of the user). The operating phase cost is referred to as the cost of penalty for incurring errors during the execution phase. The errors in usage period create a negative impact on the product’s image. The fixed cost of testing is presented in the third cost component of Eq. (15). The fixed of testing includes the cost associated with providing the necessary resources. It is affected by the errors generated during debugging and the irregular fluctuations during the testing phase. The factor
In
The aim of the problem is to achieve the set reliability goal for the software using a minimum amount. The reliability should be greater than the required reliability. Mathematically, this constraint is given by Eq. (16).
Where
The problem P1 defined in words is replaced by Eqs (15) and (16)
where
SRGM validation
Here, the SRGM proposed in Section 3 is validated using a failure dataset of faults observed in the computer with testing time to estimate the value for the parameters of the logistic testing coverage-based SRGM given in Eq. (11). The dataset used for validation purposes is of Tandem Computers (Wood (1996)). The process is performed using a statistical tool called SPSS. The description of the failure data can be referred from Aggarwal, Anand [54]. There are four releases in the dataset. Release 1 was tested for 20 weeks and 100 faults were observed. Release 2 was tested for 19 weeks and 120 faults were observed. Release 3 was tested for 12 weeks and 61 faults were observed. Release 4 is tested for 19 weeks and 42 faults are observed. The results of parameter estimation are provided in Table 2.
Estimated parameters
Estimated parameters
From Table 2 we observe the little presence of noise during the debugging process. The findings reveal that the created model accurately captures the software’s failure behavior. The data from the fourth release are utilized to solve P2 and determine the best release time under given constraints assuming that the prior version is available in the market. Problem P2 seeks to minimize the cost involved in the software development process while maintaining the needed reliability level of software, taking into account other parameters that significantly influence the debugging process. The last column of the table shows that the model fits the data for all releases. As
After validating the SRGM, the next step is to determine the optimal variables. The P2 problem is discussed in Section 3.2 and is answered using GA in MATLAB. The GA is commonly employed in the field to find the appropriate release timing [32, 57, 58]. Now, to solve P2, we assume the cost values as Cost
On solving P3 we obtained 110 weeks as the optimal time for the fourth release. This implies that software needs to be under testing for 110 weeks to achieve 90% reliability. This target has been achieved by utilizing 1649 units of resources. If the software is tested even after 110 weeks we will be able to achieve higher reliability levels. Figures 1 and 2 depict the development cost and reliability curves respectively for the software.
Sensitivity results of parameter
Sensitivity results of parameter
Cost curve.
Reliability curve.
From Fig. 1 we can observe that initially, the cost required is very high but as the testing progresses the cost keeps on going down. The requirement of 90% reliability is fulfilled after 110 weeks of testing. Similarly, after observing Fig. 2 we conclude that 100% reliability for the software can be obtained if testing continues for prolonged durations. In practical situations, the testing cannot be continued for infinite duration because with time the cost of identifying fewer faults and removing them from the software takes more effort, time, and cost.
From the cost model, it is clear that the parameter
Conclusions and scope for future research
The study investigated the effect of unbalanced fluctuations on the optimum release time and software development costs. An SRGM has been created for this purpose, which is based on the logistic testing coverage-based SRGM that includes the error generation phenomena. The model was verified using a real-life defect dataset from Tandem computers. The
Footnotes
Acknowledgments
This research was supported by an FRP grant received from the Institution of Eminence, University of Delhi, India (Ref No. IoE/2023-24/12/FRP).
