Estimating Productivity of Software Development Using the Total Factor Productivity Approach

1. Introduction

In economics, a producer is defined as an agent transforming a set of inputs into a set of outputs and the production process is defined by the production function. A similar concept is used in engineering, where a process is defined as a set of interrelated tasks that, together, transform inputs into outputs [1]. The design, control and optimization of engineering processes generally require determination of performance measures such as efficiency or productivity. However, many models of measuring process performance have usually been derived from empirical observations and have found to be inaccurate.

Let's take software development processes as an example. Several past studies, such as [2, 3] or [4] confirmed a fairly large magnitude of relative errors of the estimates of the models. In the meanwhile, the similarity between engineering processes and economic production processes has been noticed by some researchers. Hu, Plant and Hertz [5] proposed a model for software cost estimation based on the economic production theory which proved to be more accurate than traditional COCOMO [6] or SLIM [7] cost estimation models.

Measuring productivity goes beyond cost estimation since it captures not only costs, but also revenues. Total factor productivity (TFP) is a method of measuring overall productivity of economies, industries or individual firms. However, since productivity is not a property which is limited to aggregated entities, TFP could also be used on a lower level of abstraction, in business or engineering processes which consist of a series of activities.

The aim of this article is to propose the use of total factor productivity methods in the productivity evaluation of software development processes. In the first part of the article, we define productivity and its measurement and we introduce total factor productivity and indexes commonly used in measuring TFP changes. We also discuss the major data issues in the TFP approach. Consequently, we apply this approach on the process of software development. In the final part, we suggest a TFP model of software development productivity analysis.

2. Total factor productivity and its measurement

Traditionally, productivity is defined as a ratio of output over input. The total factor productivity (TFP) approach takes into account all outputs and inputs of the evaluated firm. Therefore, it is necessary to aggregate the set of outputs and outputs so that productivity becomes a scalar value.

In practice, TFP change is measured by productivity indexes or productivity indicators. Indexes have multiplicative form, whereas indicators have additive form. TFP index numbers can be based either on distance function or on price aggregation. Among measures based on distance function, we can cite the Malmquist productivity index [8] and the Hicks-Moorsteen productivity index [9]. The most common TFP measures which are based on price aggregation are the Törnqvist productivity index [10] and the Fisher productivity index [11]. The indexes based on distance functions are rather theoretical and require larger sets of cross-sectional data. By contrast, the cost-based indexes are quite straightforward to calculate and necessitate a minimum of only two observations.

Indexes are a common tool in economics to measure price or quantity changes between two periods, the most known examples being the consumer price index (CPI) or Dow Jones stock index. Since in TFP calculations we deal with the ratio of output and input quantities, we will use quantity indexes.

The calculation is based on observed or estimated quantities and prices of input and output factors. In the following text, let x = (xI,x2,…,xM) denote the input quantities, let y = (y1,y2,…,yN) denote the output quantities, w = (w1,w2,…,wM) is the vector of input prices, p = (p1,p2,…,pN) is the vector of output prices, M the number of inputs and N the number of outputs.

Let us first introduce the Fisher index of productivity [11]. It is a geometric average of the Laspeyres and Paasche indexes.

The Laspeyres index weights the quantities with the prices of the basic period. We can specify the Laspeyres output quantity index or input quantity index, respectively as

Y L = ∑ n = 1 N p n , t y n , t + 1 ∑ n = 1 N p n , t y n , t , resp . X L = ∑ m = 1 M w m , t x m , t + 1 ∑ m = 1 M w m , t x m , t

(1)

The Paasche index weights the quantities with the prices of the current period. The Paasche output (input) quantity index can be specified as

Y P = ∑ n = 1 N p n , t + 1 y n , t + 1 ∑ n = 1 N p n , t + 1 y n , t , resp . X P = ∑ m = 1 M w m , t + 1 x m , t + 1 ∑ m = 1 M w m , t + 1 x m , t

(2)

Since productivity is defined as a ratio of output over input, it is possible to define the Fisher index as the ratio of geometric averages of Laspeyres and Paasche output and input indexes, so

Π F = Y F X F = Y L Y P X L X P

(3)

The Törnqvist index of productivity [10] is defined as a ratio of output quantity index Y_T and input quantity index X_T. It is a weighted geometric average of quantity relatives. Usually, the two quantity indexes are specified in their logarithmic form as

ln Y T = 1 2 ( ∑ N [ p n , t y n , t ∑ N p n , t y n , t + p n , t + 1 y n , t + 1 ∑ N p n , t + 1 y n , t + 1 ] ln y n , t + 1 y n , t ) ln X T = 1 2 ( ∑ M [ w m , t x m , t ∑ M [ w m , t x m , t ∑ M w m , t x m , t + w m , t + 1 x m , t + 1 ∑ n w m , t + 1 x m , t + 1 ] ] ln x m , t + 1 x m , t )

(4)

Finally, the Törnqvist index of productivity can be specified as

Π T = Y T X T = e ln Y T - ln X T

(5)

If the prices are not directly observable, it is possible to approximate them using expert estimates or econometric methods such as non-linear regression. Fisher and Törnqvist indexes have several interesting properties, because of which they are classified as exact and superlative indexes (more on this subject in [12]). If the production is represented by a translog function, the Törnqvist index approximates the ideal Malmquist index and similarly, if the production is represented by quadratic function, the Fisher index approximates the ideal Malmquist index. Accidentally, a study [13] found that translog and quadratic cost functions are more suitable for representation of software production processes than the traditionally considered linear or Cobb-Douglas functional forms.

The above defined indexes are suitable for measuring and analysing productivity of a single software development process. When comparing productivity across multiple subjects or processes, the indexes have to satisfy the property of transitivity. However, the Fisher and Törnqvist indexes themselves are not transitive. An example of procedures of the conversion of non-transitive indexes into transitive indexes is the procedure proposed by Caves, Christensen and Diewert (CCD) [8] and is well described in [14]. If I denotes the number of compared processes and I_T^ij denotes the Törnqvist index for pair i,j, then the transitive indexes can be obtained as

I i j C C D = ∏ k = 1 I ( I i k T × I k j T ) 1 / I

(6)

3. Data issues in measuring TFP changes

The measurement of productivity using the TFP approach involves several data issues which substantially affect the results.

3.2 Measurement of Inputs

Another challenging task is to define and measure accurately the inputs and their prices. Traditionally, the economic theory takes into account at least the following categories of input factors: labour (T), capital (K) and materials (M). Sometimes, within materials, energy (E) and services (S) are also considered, and all these factors together are referred to as KLEMS.

Labour (L) is most often measured by the number of employees or man-hours, which should be corrected, since outsourcing of activities can distort the results. Moreover, it is preferable to distinguish among the people according to their skills, education and experience, since more skilled employees contribute to the productivity growth to a greater extent. Because of these difficulties, labour is sometimes incorporated into operating expenditures (OPEX) which are taken together as an aggregate measure of labour and materials. However, labour costs often represent the major portion of total input costs and therefore, this input should be treated carefully in order to obtain reliable TFP estimates. Wage deflators can be taken into consideration to capture the effects of wages inflation.

However, the most contentious input factor is the measurement of capital (K). The capital input should reflect the total service flow from capital assets used in the process. The assets can be of a tangible or intangible nature such as computers, software in IT, chemical reactors in chemistry production processes, transport equipment, heavy machinery, etc. Of course, the set of assets will vary a lot across industries and even company departments. The capital can be measured directly, in physical units, or indirectly in money value. Both approaches have their advantages and disadvantages; they are discussed in detail in [17].

The productive capital stock can be measured by the perpetual inventory method proposed by OECD [18]. If we denote the productive capital stock by K_t^P, then

K t P = ∑ τ = 0 T h τ F τ I N t - τ q t - τ , 0

(7)

where h_τ is an age-efficiency profile, tracing the loss in productive efficiency as an asset ages, taking values between 1 (when an assets is new) and 0 (when it has lost its entire productive capacity). F_τ is a retirement function that quantifies the share of assets of age τ that are still in service. This function is declining and takes values between 1 (when all assets are in existence) and 0 (when all assets have been retired). IN_τ is the nominal investment expenditure on the asset at time τ, which is deflated by an investment price index q_t-τ,0 where subscripts indicate a price index for the asset of age zero (a new asset) in year τ. Following the same manual, the cost of capital (rental price) can be determined using the following formula:

μ t = q t ( r t + d t ) - ( q t - q t - 1 )

(8)

where q_t is the market price of a new asset, d_t is the depreciation rate and r_t is some measure of the cost of financial capital such as the market rate of interest. However, it is possible to approximate the cost of capital inputs using regression methods following [17].

4. Productivity in engineering applications: software project management

Software development is clearly an example of process transforming a set of inputs into a set of outputs. However, some past studies, such as [19], found this process to be notorious due to cost overruns and time delays. Many scientists have tried to measure productivity of software development projects. In conformity with the traditional definition of productivity, the various methods compare the ratio of outputs and inputs. Generally, however, the methods do not take into account all the outputs and all the inputs which are involved in the analysed process as required by the total factor productivity approach.

Recently, a study [20] summarized the methods of measuring software development team productivity. Most of the authors ([21, 22], [23, 24]) considered only very simplistic measures of outputs and inputs.

For instance, the output is traditionally represented by lines of code (LOC) or function points (FP). But this simple measure does not capture either quality or other possible outputs, such as reusability of the produced code, its complexity, etc. which are not evident at first sight.

Similarly, the set of inputs is often reduced to man-hours of labour. Such a measure is nothing less than a simple labour productivity – a partial productivity measure; furthermore, these man-hours are typically not differentiated according to the knowledge and skills of development team members.

In addition, capital, materials and services are generally omitted from the productivity analysis, but development software – a typical intangible asset which is subject to amortization – represents an important input of the process; the choice of programming language as well as programming techniques clearly affect not only the number of LOC or FP, but also their quality. Usually, depreciation of hardware resources and other materials, such as energy, are considered to be negligible.

We believe that these factors could represent a source of inaccuracy of the measurement; the negligence of quality of outputs, as well as minor inputs, possibly biases the productivity estimates. The total factor productivity approach takes into account all outputs and all inputs.

Some more recent methods described in [20] take into consideration more sophisticated sets of outputs and inputs. Besides mathematical models based on regression techniques (see for example [25]), in their more recent works (i.e., [26, 27] or [28]) the authors have included the quality aspect, as well as tools and other resources.

5. TFP model of software development projects

TFP indexes can easily be estimated using common computing tools such as MS Excel or MATLAB. Fi this section, we propose a model of software development project productivity measurement based on the TFP approach.

5.3 Output and Input Prices Determination

The prices of inputs and outputs are likely not to be directly observable. In this case, it is necessary to estimate them. As mentioned before, the estimation involves either arbitrary judgment or econometric methods. In our model, we follow [17] and use both methodologies.

The prices of inputs can be estimated using the former approach. Labour costs, materials and service value are usually easy to measure, as well as the total revenue of the project. The input weight given to labour is simply the ratio of the labour costs to total revenue of the project, as well as the materials and services ratio. The aggregate weight of the rest of inputs is given by one minus labour share minus materials and services share. This aggregate weight is then divided by the approximate shares of the capital categories in the total value of the assets.

The total project costs are then divided according to the weights to the input categories. By dividing the category costs by the number of appropriate units of input, we determine the vector of costs of inputs w.

Once the costs of inputs are calculated, it is possible to estimate the costs of outputs using regression methods. Following [17], we use the Leontief multi-output cost function which takes the form

C ( y t , w t , t ) = ∑ i = 1 M w i t [ ∑ j = 1 N ( a i j ) 2 y j t ( 1 + b i t ) ]

(9)

where M denotes the number of inputs, N denotes the number of outputs, w_i is an input price, y_i is an output quantity and t is a time trend representing technological change (the number of year). Using the Shephard's lemma known in economics we can derive the input demand equations

x i = ∑ j = 1 N ( a i j ) 2 y j t ( 1 + b i t )

(10)

By applying a standard non-linear regression to these equations we obtain the input-output coefficients a_ij and the coefficients b_i. Now we can estimate the output weights as

h j t = ∑ i = 1 M w i t [ ( a i j ) 2 y j t ( 1 + b i t ) ] ∑ i = 1 M w i t [ ∑ j = 1 N ( a i j ) 2 y j t ( 1 + b i t ) ]

(11)

By dividing the total revenue according to the weights and number of outputs, we determine the costs of outputs p.

6. Conclusion

Reliable estimation of the productivity of software development process is an important task for software engineering practitioners and academics. The total factor productivity approach is a well-grounded and established methodology frequently used in economics. It is most often used at the aggregate level; however, TFP is not limited to whole economies or industries, but it can also be used in assessing productivity of companies, departments and business processes. In this article, we proposed a model of software development project productivity measurement based on the TFP approach. In the first part of the article, we defined total factor productivity and its measurement. We suggested the use of the Törnqvist and Fisher quantity indexes to measure the TFP changes across time periods. When comparing multiple software development processes, the transitive versions of the indexes should be used.

In the following section, we dealt with the major data issues which have to be taken into consideration when measuring inputs, outputs and their prices.

Consequently, we discussed the issues of measuring productivity in engineering applications, in particular in software project management. We found out that most of the currently used methods are in fact simple labour productivity measures and do not include all inputs and outputs that participate in the development process.

In the final part, we suggested a TFP model of software development productivity analysis.

In order to capture all the relevant outputs and the quality, we proposed including the rate of code reuse, the code complexity, functionality and length. Furthermore, we suggested considering other outputs of the software development process, namely data produced, documentation and training of persons.

Concerning the inputs, we proposed to differentiate the labour input according to the function of the development team functions: at least, engineering, testing, management and support labour force should be included in the analysis. Besides labour, we prefer to consider other input factors: materials and services, tangible and intangible assets, and other capital. All these factors together reflect the complete set of classical input factors considered in economic theory.

Finally, we described the process of the determination of input and output prices in case they are not directly observable. Both expert estimates and regression methods can be used.

An accurate estimation of productivity changes could help software engineers identify strengths and weaknesses of development processes, and improve the useful output of the process which, in turn, could be accomplished with less effort. However, empirical evidence is needed, which could be the focus of the future research on this subject.