A predictive model to identify Kanban teams at risk

Abstract

Kanban, which is an agile process methodology as well as a means to implement lean principles, has been growing as a project management framework across a range of domains, including manufacturing, software development and data science. This paper explores, for teams using Kanban, the ability to predict low team performance. The prediction is based on an analytical model that uses specific project metrics that can be collected via the team’s visual Kanban board. Specifically, data from 80 teams was used to build and test machine learning models that predict teams at risk for delivering low quality results. The model developed was significantly better than the baseline situation of thinking that all teams were at risk. While this analysis was done within a data science project context, the results are likely applicable across a range of information system projects.

Keywords

Kanban metrics project management team performance data science project management

1. Introduction

Studying and predicting team performance and teams at risk of poor performance has been studied for decades in both academia (Minaei-Bidgoli et al., 2003; Kabakchieva, 2013; Osmanbegović & Suljić, 2012; El-Halees, 2008) and industry (Wolf et al., 2009; Levesque et al., 2001). In both academia and industry, the goal of identifying teams at risk of poor performance is to reduce the risk of the poor performance by allowing the team leaders (e.g., managers or classroom instructors) to provide appropriate advising in a timely manner (Minaei-Bidgoli et al., 2003). One way to help identify teams at risk is via the development of a predictive model, which would learn, from previous projects, the characteristics of teams that did not do well.

However, the factors impacting a project’s success or failure are dependent on the project’s context (Shenhar et al., 2002; Dvir et al., 1998). Hence, it is not surprising that it has been noted that more research is needed to adapt a more project-specific approach to identify exact causes of a project’s success or failure (Shenhar et al., 2002; Dvir et al., 1998). One example of a context that could impact the factors driving a project’s success is the project management framework used by the project team.

Agile process methodologies are a commonly used project management framework within information system projects. Specifically, agile frameworks are a group of incremental and iterative methods that have been shown to be effective in helping to manage projects. During the past 25 $+$ years, these frameworks have become increasingly popular for software projects because they enable lightweight processes that accept change and can deliver a project via rapid iterations (Pressman & Maxim, 2015).

Kanban, a process framework that focuses on visualizing the flow and minimizing work in progress, is an agile approach that aims to streamline the amount of work done at the moment. Kanban has been growing in use across a range of domains such as software development (Kirovska & Koceski, 2015) and data science (Saltz et al., 2017). However, to the best of our knowledge, for teams using Kanban, there have been no studies on how to predict projects at risk of delivering poor results.

This study aims to help address this gap by collecting easily obtained Kanban metrics and then building a predictive model that identifies teams at risk for delivering low quality results. In short, the research questions address by this research are:

RQ1. What are the possible metrics used to evaluate Kanban projects?

RQ2. Based on project metrics collected during the project, can one predict which teams are at risk to perform badly?

The rest of the paper is organized as follows. First, a review of project metrics is presented, which is followed by a discussion on the key Kanban concepts as well metrics specifically designed for Kanban. Next, a classification model is developed to help identify teams at risk. Finally, a synthesis of our observations will be provided within our discussion and conclusion.

2. Background

2.1 Project management and project metrics

Traditionally, due to the multi-attribute nature of a project’s success, the success of a project is assessed using multiple measures, including internal (meeting goals, schedule, budget) and external (project’s impact on its customers, and on the developing organization itself) measures (Lipovetsky et al., 1997). These measures are often called as metrics or dimensions (Lipovetsky et al., 1997).

According to (Fenton & Pfleeger, 1998), teams use metrics to understand, control and improve what is done and how it gets done. Common needs for metrics are related to supporting communication and decision making. Pulford et al. (1995) give us the following motivations for metrics use:

•
Project planning and estimation;
•
Project management and tracking;
•
Understanding quality and business objectives;
•
Improved software development communication, processes, and tools.

However, metrics are only valuable if one can use them. Specifically, good metrics have three key attributes: the data is consistent, inexpensive to collect, and quick to collect (Bladt & Filbin, 2013). With this in mind, in a literature review several categories of metrics were identified (Kupiainen et al., 2014) including: Iteration Planning, Iteration Tracking, Motivating and Improving, Identifying Process Problems, Pre-release Quality, Post-release Quality. Some of these categories could be useful for predicting project success. Specifically, process problem metrics, which are metrics that are used to identify or predict problems in order to solve or avoid them, has been extensively studied (Petersen & Wohlin, 2011; Trapa & Rao, 2006; Mahnič & Zabkar, 2012; Petersen & Wohlin, 2010; Shen & Ju, 2007; Mujtaba et al., 2010; Tudor & Walter, 2006). However, none of these studies explored Kanban related metrics. Perhaps the closest relevant study was when Catal & Diri (2009) explored software fault prediction. Although this effort focused on risk prediction, it did not explore this prediction challenge within a Kanban context. In fact, to the best of our knowledge, there are no studies that have focused on predicting a projects’ success based on Kanban metrics.

A Gantt chart is a type of bar chart that illustrates a project schedule as well as dependency relationships between activities and current schedule status. It is a common way to structure and visualize work. The critical path of a project is a key project management concept when using Gantt charts. It is “the sequence of scheduled activities that determines the duration of the project” (PMI, 2004). In other words, it is the longest sequence of tasks in a project plan that must be completed on time in order for the project to meet its deadline. If there is a delay in any task on the critical path, then the entire project will be delayed. One can view the critical path method as a step-by-step project management technique to identify activities on the critical path. It is an approach to project scheduling that breaks the project into several work tasks, displays them in a flow chart, and then calculates the project duration based on estimated durations for each task.

While Gantt charts (in conjunction with critical path analysis) have often been used to help manage projects, for better and worse, they display the most detail and complexity with respect to the relationship among the project tasks (Landry & McDaniel, 2015). It has also been noted that Gantt charts focus on providing visual effects, but contain no quantitative information to help monitor a software project (Zhang, 2011). Furthermore, with respect to critical path analysis, when using Kanban, the focus is not on the overall critical path, but on limiting work-in-progress and bottlenecks, which can cause tasks to not be completed. As described in the next section, this is done via the use of the Kanban board and defining a maximum number of work-in-progress tasks. When using Kanban, when a task becomes a bottleneck, the entire team then focuses, if appropriate, on eliminating the bottleneck. This suggest why, in contrast to Gantt charts, Kanban is increasing in popularity (Al-Aidaros & Omar, 2017). For example, it has been noted that “Gantt charts and reports that looked solid on paper often have failed to deliver the software on time” (Sutherland, 2004) and that many companies are switching from Gantt chart-based planning to other, more agile methodologies (Moore et al., 2007).
2.2 Reviewing kanban and the growth in its usage

Kanban is Japanese for “visual signal” or “card” (Ohno, 1988; Sugimori et al., 1977). Starting in the 1940s, Toyota line-workers used Kanban (with physical cards) to improve their manufacturing process. The system’s highly visual nature allowed teams to communicate more easily on what work needed to be done. In the first academic study about Kanban (Sugimori et al., 1977), three reasons for its use were proposed: reduction in information processing cost, rapid and precise acquisition of facts, and limiting surplus capacity of preceding shops or stages.

Based on a systematic mapping study (Ahmad et al., 2018), the most common definition of the Kanban methodology is the definition defined by Anderson (2010, p. 6): an “evolutionary change method that utilizes a Kanban pull system, visualization, and other tools to catalyze the introduction of Lean ideas…the process is evolutionary and incremental”. More specifically, Kanban is based on three key principles:

•
Visualize the workflow – Split the work into pieces; write each item on a “card” and put on the “wall” and using named columns to illustrate where each item is in the workflow. By creating a visual model of work and workflow, the team can observe the flow of work moving through its Kanban system. Making the work visible—along with blockers, bottlenecks and queues—instantly leads to increased communication and collaboration.
•
Limit WIP (work in progress) – assign explicit limits to how many items may be in progress at each workflow state. By limiting how much unfinished work is in process, the team can reduce the time it takes an item to travel through the Kanban system. The team can also avoid problems caused by task switching and reduce the need to constantly reprioritize items.
•
Focus on Flow By using work-in-process (WIP) limits – by developing team-driven policies, the team can smooth the flow of work and make sure the team is focused on getting work completed.

Limiting the amount of work-in-progress, at each step in the process, prevents overproduction and reveals bottlenecks dynamically and is one of the key differences between a Kanban board and any other visual storyboards used within other methodologies. Note each Kanban board has columns that the team collectively determines what are the most appropriate columns for the team to use. As shown in Fig. 1, columns often include “to do”, “doing”, “validate” and “done”. One can see that for the project shown in Fig. 1, the project did not have any bottlenecks. However, if the “validate” column, for example, reached the work-in-progress limit, other work that was finished in the “doing” phase could not be moved into the “validate” column. Hence, the team would understand that there is a bottleneck and would collectively work to help ensure that at least one of the validate tasks gets completed (and hence, moved to the “done” column, thus enabling another task to be moved to the “validate” column).

Figure 1.
Example Kanban board.

Note that many teams now use an e-version of the Kanban board (Nakazawa & Tanaka, 2015, 2016; Ostergaard, 2016), and Fig. 1 shows an e-version created using a web-based tool known as Trello.

With respect to information systems development, it has been shown that Kanban has a positive impact on project results (Lei et al., 2017). For example, at BBC Worldwide (Middleton & Joyce, 2012), the lead time to deliver software improved by 37%, the consistency of delivery rose by 47%, and defects reported by customers fell 24% as compared to the previously adopted agile method.

The value of Kanban has also been observed in other information system contexts, such as within a data science project where a Kanban based approach has been shown to be a more effective framework, when it was compared to other approaches (Saltz et al., 2017). Furthermore, although Kanban is often used within information systems context, the Kanban method has been applied to other aspects of knowledge work (Leybourn, 2013), such as Human resources (Benson, 2011) and recruitment (Lambert, 2014). Overall, it has been noted that Kanban offers improved project visibility, quality, team motivation, communication and collaboration (Ahmad et al., 2013, 2016). Hence, it is not surprising that the use of Kanban is growing. For example, in the latest “State of Agile Report” (VersionOne, 2016, 2018), the use of Kanban, as an agile technique, grew to 65% (in 2017) from 50% in 2016, and from 31% in 2014. In short, the use of Kanban is growing quickly.
2.3 Kanban metrics

In a literature review focusing on Kanban projects (Kupiainen et al., 2015), none of the identified studies explored predicting projects success using Kanban-based metrics. There are, however, several commonly used metrics to help track the progress of a Kanban project. These metrics might also be leading indicators of the overall performance of the Kanban team (Power, 2014). Below, we review the most frequently used metrics and explain the purpose of the metric.

Lead time: Lead time is the total elapsed time from when the work item was started until it is declared complete and accepted by the customer. This is a key measure of the team’s throughput and productivity (Mahnič, 2013). Lead time measures duration from beginning to end. This includes process time, as well as time that work spends sitting in queues, or wait states. By tracking lead time metrics over a set period of time, one can determine the impact of any changes one makes – if the change helps deliver value faster, or if the change gets in the way of delivering value.

Cycle time: Cycle time measures how long it takes a work item to get from point A to point B. Since cycle time can be measured from any two starting and ending points on a Kanban board. It’s common for several categories of cycle time to exist on one board (e.g., deployment cycle time, development cycle time, QA cycle time, etc.). Lead time and cycle time are similar and easily confused; both help the team understand how long work takes to flow through our value streams, but they measure different segments of the process.

Work in process (or work in progress) – WIP length: In general terms, “WIP” refers to any work that has been started but is not yet providing value to the customer (i.e., work that is not yet “done”). In short, it is all the work that is actively being worked on at any one time. Tracking work items that are started, but not yet finished, can help the team improve the overall flow of value through the system. Practically speaking, work cannot add value to the customer, team, or organization unless it’s finished work. In addition, a team with fewer WIP items is more agile, since the team can get feedback more frequently with respect to completed tasks and which new item to start.

Queue length: Queues form in a team’s process when work waits between different stages (or columns on the Kanban board). Since queues often represent the majority of a work item’s total life cycle, it’s important to understand how they affect the team and where they occur in a team’s process. Limiting the time that work spends in queues can help reduce the overall cycle time and keep work flowing through the system. An efficiency diagram measures the difference between total WIP and the work that’s waiting in queues. It illustrates when the work in queue is growing as a percentage of the total WIP. This allows one to pinpoint where work is likely stuck in a queue and also investigate what can be done to get work flowing again.

Number of blockers: Kanban systems often use a “blocker” symbol to visually indicate work that cannot move forward in the process. Different from work in a queue – which is often simply waiting its turn to be pulled into process – a blocker is typically waiting on an external dependency or some failure condition. Blockers double as a useful signal that a piece of work needs immediate attention as well as a valuable flow metric. Blockers are one of the easiest board elements to measure, especially for new teams that lack the work history necessary for other metrics. Simply stated, count how many items are blocked and record how long those work items stay blocked.

Throughput: Throughput is the average number of units processed per time unit. In a Kanban system, examples can include “cards/tasks per day” or “cards/tasks per week”.

Figure 2 shows a cumulative flow diagram (CFD), which visualizes two of the most frequent metrics (WIP and cycle time).

Figure 2.

Cumulative flow diagram for WIP and cycle time (Scotland, 2009).

3. Methodology

Following the findings from Lipovetsky et al. (1997) and Tishler et al. (1996), which suggest that managerial activities should emphasize managerial variables that can maximize the benefits to the customer (via meeting the design goals), this study explores metrics that can help predict successful project outcomes (i.e., deliverables to the customer).

To investigate the predictive power of existing and potentially newly devised Kanban metrics, 80 Kanban projects were analyzed. Specifically, metrics were collected for 80 student teams in a master’s level data science course, across four semesters. The instructors acted as the customer for each team, and each team worked on a semester long data science project using the Kanban process methodology, including the use of an online Kanban board tool (www.trello.com).

3.1 The use of students and their applicability to industry

The generalization to industry, when using students, has often been criticized (Runeson, 2003; Sjøberg et al., 2002, 2005; Salman et al., 2015). This is due to the fact that in-class tasks are often far from reality, and therefore, the task is not representative of tasks in industry. This is one of the key reasons the results of an in-class experiment often do not transfer to industry (Sjøberg et al., 2005).

More generally, some think that using students is an external threat to validity (i.e., when the researcher generalizes beyond the groups in the experiment to other groups not under study). However, based on additional analysis of the literature, the results from students can be applied to an industry context, if the use of students is done in an appropriate manner. Within this research context, we first note that the students worked on a semester long project, which was defined in such a way to be as realistic as possible. In addition, master students, with an average of 3 years of IT experience, were selected as participants. The use of graduate level students ensures that the subjects had more advanced information systems knowledge, and as noted by many, constitute a good sample of the next generation of professionals entering the job market (Taibi et al., 2017; Runeson, 2003; Kitchenham et al., 2002; Tichy, 2000; Salman et al., 2015).

Furthermore, many studies have not found a significant difference in quality of the code produced by students as compared to the quality of code produced by professionals (Höst et al., 2000; Runeson, 2003; Berander, 2004; Svahnberg et al., 2008; Salman et al., 2015). For example, in Salman et al. (2015), professionals produced better code than students only when they were already familiar with the tasks to be done.

Finally, classifying experimental subjects (students) by their status (student or not student) is a proxy for more important and meaningful classifications, such as classifying the participants according to their abilities and experience, and effort should be invested in defining and using these more meaningful classifications (Feitelson, 2015).

Hence, within our context of a new realistic project task, graduate student teams are a good proxy for teams in industry. In short, one can view our analysis focusing on junior professionals with, on average, three years of IT experience.

3.2 Project context

As part of the introductory data science course, students were required to work on a group project, which started early in the semester and continued until the end of the semester. The project was twenty-five percent of the course grade; thus, the students were highly motivated to work on the project. The project was done using the R programming language, a popular data science tool that is used in both industry and academia. The analysis required the team to perform many typical data science tasks, such as data cleaning and leveraging machine learning algorithms to deliver insight to their client.

Specifically, the project was positioned in a way that the students were to act as consultants and analyze a large data set of customer survey responses for their client. In order to have the project be as realistic as possible, specific instructions were not provided to the students (on what analysis was to be done). The goal for each team was to identify and then answer “interesting” questions, such as understanding how customer satisfaction varied across surveys (frequent vs non-frequent customers, etc.). This goal was the same for all teams (i.e., to get actionable insight from the data so that the organization could improve customer satisfaction). For example, if customer service was observed to be an important driver for improving customer satisfaction, a team might suggest that the organization should improve customer service by providing extra services to gold members.

All teams worked on well understood datasets, and as such, all the teams had the same level of difficulty with respect to working on the project. Note that no specific questions/goals were provided to any of the teams. For example, what to analyze was determined by each team, and was a function of how the team determined what might be useful, what was possible with the data that was provided as well as project duration. In other words, teams tried to identify important drivers (factors) for improving customer satisfaction. In doing the projects, each team needed to create tasks such as “data preprocessing: N/A handling” or “find most important factors that impact frequent customers”. To identify the appropriate questions, each team explored the data available as well as discussed possible analysis options with their client (or more accurately, a person acting as the client). The dataset was a modified version of a real dataset of survey responses. Hence, the data was not real, but was representative of the actual challenges one might face in executing a data science project. For example, some values in the dataset were blank. This reflected a typical ‘real-life’ challenge in how to handle missing values that was due to the fact that some of the surveys asked more questions than other surveys (i.e., there were different survey given to different customers, some surveys asked more questions than others).

Finally, students had access to a description of each column of the dataset. The attributes included information about the person who responded to the survey (e.g., place of residence, a member of their rewards program, and if so, what level), information about the service (e.g. location) and information about the responses to the survey from the customer (e.g., would they recommend the service/company to a friend).

3.3 Data collection

As shown in Table 1, 362 students participated in the study across 80 teams during four semesters.

Table 1
Number of teams and students participated in the experiment

	Spring ${}^{\prime}$ 17	Fall ${}^{\prime}$ 17	Spring ${}^{\prime}$ 18	Fall ${}^{\prime}$ 18	Total
Number of students	29	144	78	111	362
Number of teams	6	32	19	23	80
Average team size	4.9	4.5	4.1	4.8	4.5

Each team used a web-based Trello board, which can be used as a virtual Kanban board. The tasks were managed by the students (i.e., the team members). Each of the team’s activities (such as task moves, adding comments, etc.) was logged within Trello and retrieved using the Trello API. Specifically, as part of this research effort, an R script was developed to pull the data from Trello (using their API). Based on the data retrieved from Trello, the appropriate metrics were then calculated. These metrics were then provided as input to the machine learning (predictive modeling) algorithms.

Note that each of the metrics were evaluated three times during semester: after two weeks (i.e., near the beginning of the project), at the mid-point of the project and two week prior to the project being submitted. Therefore, the data can be thought of as an 80 $\times$ 3N matrix of data that can be used to predict whether teams are at risk or not (i.e., 80 rows and 3N columns).

In addition to the Kanban process metrics, the target variable “grade”, was also part of the dataset for each team. The grade was used to identify teams with poor performance. The grades were assigned to the teams by two faculty members independently, and evaluated after the projected completed. The grade was assigned to the team as a whole (i.e., same grade for all team members). To generate a score for each team, the evaluation from the two faculty members were averaged. Furthermore, the grades were scaled such that approximately half the student teams were deemed “at risk” of a poor result. Specifically, grades were scaled such that grades were out of 20, and teams that scored less than 19 indicated that a team was at risk. Forty-six percent of the teams (37 of the 80 teams) were deemed “teams at risk” (i.e., had a grade less than 19), and fifty-four percent of the teams (43 of the 80 teams) were deemed not at risk for poor performance (i.e., the project was deemed a success). In other words, the grades were used as an estimation of project quality, and the concept of risk in the experiment was with respect to low quality. Hence, the predictive algorithms could use the “grade” or “at risk” variable to use as their dependent variable.

3.4 Existing and new metrics

The following, previously defined, Kanban metrics were used:

•
Lead time, WIP, blockers;
•
Throughput – average number of units processed per time unit. In a Kanban system, examples can include “cards per day”, “cards per week”, or “story points per iteration” – in our case we can have “X per update”;

However, other existing metrics were not used:

•
Queues – for this project, queues were not used;
•
Cycle time (how long it takes a work item to get from point A to point B) – it was not possible to calculate in a consistent manner, since column the names differed from team to team.

In addition to previously defined Kanban metrics, several additional metrics were also calculated and evaluated:

•
Number of tasks: Total number of tasks on the board (in any state);
•
Number of actions: Total number of transactions by the time of measurement; this include commenting, moving card, renaming card headers or adding description;
•
Number of completed tasks (in “done” column): Helps to understand real progress of teams;
•
Number of tasks after “to do”: Tasks in progress and completed tasks;
•
Number of students who created or edited cards: This metrics shows how equal work is distributed across team members;
•
Number of words in “done”: The idea is that very short task descriptions might be not that valuable in contrast to tasks with more detailed information.

3.5 Model evaluation

The goal of the analysis was to propose metrics (i.e. features for ML models) that have high predictive power (i.e. that can distinguish team at risk from other teams). As previously noted, teams “at risk” means teams that are likely to show low performance by the end of semester (i.e. will get low grade).

In order to analyze effectiveness of the models, a baseline solution is needed. However, since there has been no predictive model generated and published previously, the baseline is defined as the task of assigning one label to all objects:

a)
Assign “at risk” to all objects (majority vote rule, maximizing accuracy).

–
Confusion matrix $\begin{pmatrix}37&0\\ 43&0\end{pmatrix}$ .

b)
Assign “not at risk” to all objects (since we do not want to overlook teams at risk).

–
Confusion matrix $\begin{pmatrix}0&{37}\\ 0&{43}\end{pmatrix}$ .

Table 2
Evaluating metrics with the baseline approach

All teams are “at risk” All “not at risk”

Recall or sensitivity (true positive rate) 0.46 0 – undefined

Precision (positive predictive value) 1 0

Selectivity or specificity (true negative rate) 0 – undefined 0.54

Accuracy (% correct) 0.46 0.54

F1 (harmonic mean of precision and recall) 0.7 0 – undefined

Sensitivity and selectivity (harmonic mean) 0 – undefined 0 – undefined

Below, we review typical approaches that might be used to evaluate different models within this project context. Table 2 shows the value of each metric for each of the two baseline approaches.

•
Sensitivity/recall/True Positive Rate (TPR) $=$ TP/P; the proportion of actual positives that are correctly identified as such (e.g., the percentage of teams “at risk” which were correctly identified as having this condition);
•
Precision/Positive Predictive Value (PPV) $=$ TP/(TP $+$ FP); the fraction of relevant instances among the retrieved instances (e.g. percentage of “teams at risk” among classified as such;
•
Specificity/selectivity/True Negative Rate (TNR) $=$ TN/N; proportion of actual negatives that are correctly identified as such (e.g., the percentage of “teams not at risk” which were correctly identified as not having this condition);
•
Accuracy: percentage of correct predictions;
•
F1 score: harmonic mean of precision and recall;
•
Sensitivity and specificity score: harmonic mean of Sensitivity and Specificity.

It has been noted that precision and recall should only be used in situations where the correct identification of the negative class does not play a role. This is true in information retrieval, which is where this term originated (Powers, 2011). In information retrieval, there is generally an unknown and much larger of true negatives, as compared to the actual numbers of relevant and retrieved documents, and hence, not exploring true negatives is not an issue.

In contrast to information retrieval, where number of true negatives is large and unknown, in many other applications (e.g., medicine, team performance analysis), the assumption of very large numbers of true negatives versus positives is rare (Powers, 2011). For example, in this project management context, providing support to “at risk” teams requires the use of a limited resource and providing this support to teams that don’t need support may leave other teams unsupported (or incur undue expense). Thus, in this project modeling context, precision and recall, as well as F1 (harmonic mean of recall and precision) should not be used.

In this situation (where there is not the assumption of a very large number of true negatives), evaluating a model based on both sensitivity and specificity is appropriate (Yerushalmy, 1947; Parikh et al., 2008). This is because these measures consider all entries in the confusion matrix. While sensitivity deals with true positives and false negatives, specificity deals with false positives and true negatives. Furthermore, sensitivity and selectivity are closely related to $\alpha$ (type I) and $\beta$ (Type II) errors in testing hypotheses (Sharma et al., 2009). Specificity is $1-\alpha$ , and sensitivity is $1-\beta$ . Thus, Type I and Type II errors are directly related to sensitivity and selectivity, and if one wants to ensure low Type I and Type II errors, the metric needs to incorporate both sensitivity and selectivity. For example, increasing the specificity of the test lowers the probability of type I error. Thus, specificity is a statistical measure how well a binary classification test correctly identifies the negative cases (Sharma et al., 2009). Similarly Type II error is denoted by $\beta$ .

This suggests that the combination of sensitivity and specificity is an appropriate holistic measure when both true positives and true negatives should be considered (Baron, 1994; Boyko, 1994; Pewsner et al., 2004). In terms of combining sensitivity and selectivity, one should consider both the arithmetic and harmonic mean:

•
Arithmetic mean $=$ 0.5 $\times$ (sensitivity $+$ selectivity).
•
Harmonic mean $=$ 2 $\times$ sensitivity $\times$ selectivity/(sensitivity $+$ selectivity).

One can note that using the harmonic mean puts more of a focus on the lower of the two numbers being averaged, which is why is it used in F1 and why the harmonic mean is also widely leveraged when combining sensitivity and selectivity (Hong et al., 2009), especially in medicine (Hoff et al., 2009; Hildebrandt et al., 2017). Thus, using the harmonic mean of sensitivity and selectivity was the approach used to evaluate the different models.
3.6 Modeling techniques evaluated

	All teams are “at risk”	All “not at risk”
Selectivity or specificity (true negative rate)	0 – undefined	0.54
Accuracy (% correct)	0.46	0.54
F1 (harmonic mean of precision and recall)	0.7	0 – undefined
Sensitivity and selectivity (harmonic mean)	0 – undefined	0 – undefined

In addition to linear modeling, additional Machine Learning (ML) models, listed below, were also evaluated. These models were evaluated using the python machine learning library scikit-learn (Pedregosa et al., 2011), where they were used to predict the “at risk” variable:

•
K-Nearest Neighbors algorithm (KNN);
•
SVM (with multiple kernels);
•
Gaussian process classification (GPC) based on Laplace approximation;
•
Decision Trees (DT);
•
Random Forests (RF);
•
Multi-layer Perceptron (MLP);
•
Gaussian Naïve Bayes (NB);
•
Quadratic Discriminant Analysis (QDA).

Cross-validation was used to evaluate the algorithms performance. Cross-validation is one of many similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice (James, 2013). Specifically, 5-K-Fold cross-validation and MinMaxScaler were used (transforming all numbers into [0, 1]).

However, running cross-validation once might generate unstable results. This is due to the fact that splitting the dataset into train and test parts (which is what K-fold cross-validation does) is a random process and there is no way to ensure replicability and reliability of the results. In short, if one replicates the experiment with the same parameters of models, the quality of the prediction might not be the same. In contrast, if one repeats the cross-validation, and shuffles the data before each iteration, and then takes an average of those repeated cross-validations, the results will converge to a stable mean (with a normal distribution). Therefore, we performed the 5-fold cross-validation 1000 times as demonstrated via the following logical flow:

For all tuning parameters (such as C for SVM, alpha for MLP, k for KNN) {

do 1000 times {

5-cross-validation {

calculate harmonic mean of sensitivity and selectivity

}

}

Pick the parameters with the highest harmonic mean (of sensitivity and selectivity)

}

3.7 Feature selection/engineering

It is often the case that when there are many features (relative to the training sample size), it is beneficial to reduce the number of features (Hua et al., 2005). This could be the situation with our dataset. Specifically, since the dataset being explored had many features/attributes (compared to number of observations – rows of data), it is possible (or even likely) that one could get better results if one reduced the number of features/metrics.

To do the feature selection, we employed multivariate analysis methods such as Principal Component Analysis, which has been proven to be an effective approach within a project management context (Tishler et al., 1996; Shenhar et al., 2002; Dvir et al., 2003). In short, since the dataset had only 80 observations, it was hypothesized that having fewer features (e.g., 5-10) would improve results. Therefore, the following feature selection techniques were leveraged within this analysis (with a goal of improving the predictive accuracy of the analysis):

•
Principal component analysis (PCA) – A dimensionality reduction method based on the correlation matrix. This multivariate analysis method has been shown to be effective in multiple applications (Shenhar et al., 2002).
•
Manual feature selection based on metrics of different categories – Based on exploratory metrics analysis and high correlation between some variables, one can hypothesize that some metrics can be grouped together (to avoid multicollinearity). If two numerical features are highly correlated, then one doesn’t add any additional information (it is determined by the other). For example, number of tasks after “to do” might be eliminated from the analysis if one uses number of WIP tasks, since these two metrics are conceptually very close to each other. In other words, when using this approach, the analysis will use one metric from each logical group of concepts.
•
Univariate feature selection – Select the top K features based on Chi-squared univariate statistical test (scikit-learn, 2018). The chi-squared test measures dependence between stochastic variables and the target class variable, hence using this function eliminates the features that are the most likely to be independent of class and therefore irrelevant for classification. Therefore, this score can be used to select the K features with the highest values for the test chi-squared statistic.

4. Results

4.1 Linear regression

Linear regression was first explored, but the results didn’t yield any models with high predictive value (i.e., high R-sq). For example, when trying to create a linear model using ordinary least squares (OLS) and using all the metrics, the R squared value was 0.1 and all $P$ -values for the variables were over 0.4 (i.e., not significant). Even after removing the highly correlated variables (to avoid cross-correlation), the model still did not show any significant variables (all $P$ -values were over 0.28). Therefore, there was no linear relationship that could be used to predict teams at risk, and hence, more advanced modeling was utilized, which is discussed in the next section. In other words, there could be a more accurate non-linear model (for example, a situation where low and high prices for varying levels of internet connectivity might drive low satisfaction, but moderate prices for an appropriate level of internet connectivity might drive higher satisfaction).

4.2 Comparison of predictive power of the metrics

The results, using 5-fold cross-validation, with 1000 iterations and a goal of maximizing the harmonic mean of sensitivity and selectivity, for predicting teams at risk, using KNN, SVM, DT, RF, MLP, NB, GPC, QDA machine learning techniques, ranged from 0.27 to 0.38, with KNN performing the best. These results were after tuning the key parameters of the model, such as the cost ‘C’ for SVM. Note that the average Sensitivity and Selectivity were calculated during each iteration, and then averaged (i.e., those calculations did not use the averaged recall, selectivity, and precision values). The statistical $t$ -tests ( $\alpha=$ 0.05) found that all methods significantly outperformed the baseline (i.e., their harmonic mean of sensitivity and selectivity are significantly higher than 0, $p$ -value $\sim$ 0). We note that KNN also had the highest accuracy.

4.3 Feature engineering/selection

As described in our methodology, feature engineering/selection was explored via principal component analysis, manual feature selection and univariate feature selection. The results of the feature selection analysis are presented below.

PCA: With respect to principal component analysis (PCA), an analysis was done when the dimensions were reduced to 2, 3, 4, 6 and 8. Using PCA, with four components, led to the highest improvement of the harmonic mean of selectivity and sensitivity (0.46 when using NB). We also note that with six components, the harmonic mean of selectivity and sensitivity was 0.45 (also when using NB). Table 3 has the full PCA results.

Table 3
Performance of ML models depending on number of PCA components

Algorithm	Number of components/dimension
	2	3	4	6	8
k-nearest neighbors (KNN)	0.38	0.38	0.35	0.37	0.38
Support Vector Machines (SVM)	0.09	0.1	0.27	0.28	0.24
Decision Trees (DT)	0.36	0.42	0.43	0.44	0.42
Random Forests (RF)	0.36	0.41	0.45	0.43	0.39
Multilayer Perceptron (MLP)	0.16	0.18	0.19	0.2	0.21
Naïve Bayes (NB)	0.36	0.38	0.46	0.45	0.37
Gaussian process classifier (GPC)	0.18	0.25	0.21	0.22	0.25
Quadratic Discriminant Analysis (QDA)	0.34	0.36	0.4	0.38	0.31

Manual feature selection: The attributes were put into logical groups, and the key attribute used, as noted below:

•

WIP (removed ‘Number of tasks after to do’).

•

Number of blockers (no attributes were removed).

•

Throughput (removed ‘Number of tasks’, ‘Number of tasks in done’, ‘Number of Actions’).

•

Number of words in done (removed ‘Project update evaluation’).

As shown in Table 4, the manual selection of features led to an improvement of the harmonic mean of selectivity and sensitivity to 0.45 (using NB), which is approximately the same result as was derived via PCA.

Table 4

Performance of KNN models for the four manually selected features

Algorithm	Sensitivity and selectivity (harmonic mean)
	Updates 2&3 (8 features)	Update 2 only (4 features)	Update 3 only (4 features)
KNN	0.36	0.34	0.41
SVM	0.2	0.03	0.12
DT	0.36	0.44	0.35
RF	0.35	0.37	0.35
MLP	0.13	0.1	0.1
NB	0.36	0.23	0.46
GPC	0.22	0.18	0.31
QDA	0.35	0.32	0.32

Univariate feature selection: As shown in the previous analyses, SVM, MLP and GPC performed significantly worse than other methods. Therefore, these approaches were not evaluated during univariate feature selection. As shown in Table 5, univariate feature selection generated a result with a harmonic mean of 0.49 (using KNN), which was the best model identified.

Table 5

Univariate feature selection performance

Sensitivity and selectivity (harmonic mean)	Selected metrics with the best result
	2	4	6	8
	[c]‘throughput3’,
‘num_tasks2’	[c]‘num_WIP3’,
‘throughput3’,
‘num_tasks2’,
‘num_words_in_done3’	[c]‘num_WIP3’,
‘num_blockers3’,
‘throughput3’,
‘num_tasks2’,
‘num_tasks_afterTODO3’,
‘num_words_in_done3’	[c]‘num_WIP3’,
‘num_blockers1’,
‘num_blockers3’,
‘throughput3’,
‘num_tasks2’,
‘num_tasks_done3’,
‘num_tasks_afterTODO3’,
‘num_words_in_done3’
KNN	0.48	0.49	0.40	0.40
DT	0.39	0.36	0.37	0.36
RF	0.43	0.39	0.38	0.36
NB	0.35	0.44	0.44	0.44
QDA	0.35	0.36	0.36	0.37

In summary, univariate feature selection was the best feature selection approach, and that approach outperformed the results when no feature selection was used. Specifically, using attributes such as the number of items that are in progress during update 3, the throughput that took place during the period before update 3, the number of tasks defined during update 2 and the average number of words in the description of tasks done during tasks 3, were deemed most important. The attributes, using KNN, generated a result of close to 0.5, which was much better than using all the attributes (which generated a sensitivity and selectivity harmonic mean of 0.38). However, it should also be noted that PCA as well as manual feature selection generated similar results (i.e., 0.46 for both). Hence, this suggests that the results were stable, since they were generating similar results across different approaches.

5. Discussion

Planning fallacy is a situation in which predictions about how much time will be needed to complete a future task are optimistically biased and hence, the time needed to do the task is underestimated (Kahneman & Tversky, 1979). While the planning fallacy is an important issue within project management, it is not a key issue when teams use the Kanban framework. This is because when teams use Kanban, they do not generate tasks estimates. In short, as previously mentioned, the Kanban team members work on tasks as the team puts the tasks on their Kanban board (without an explicit task estimation). Hence, this lack needing to do task estimation when using Kanban is why this study did not focus on task estimation nor the planning fallacy related to task estimation.

We also note that even if one considers the final project deadline, Kanban’s focus on minimizing work-in-progress helps each team ensure a usable result at the project due date. Specifically, with respect to information systems development (e.g., software applications, data science), at the project’s deadline, there might be some features or analysis that might not yet have been completed. However, even with everything not being completed, there is still a usable and working system. For example, with respect to the data science projects analyzed during this research effort, the goal was to have an analysis with actionable insights by the end of semester. More advanced teams might have explored additional models or improved their tuning parameters within the models that were implemented. In short, within this type of data science analysis, there is always room for improvement, and even in industry, some tasks are typically not completed due to the perceived diminished return on investment.

The study does have several limitations, which suggests potential next steps. For example, there are other forms of project risk beyond quality (e.g., budget issues or the delay of a project) and work could be done to build and predict these other types of project challenges.

In addition, while the model was far better than the default baseline, there is still significant room for improvement. One way to improve the model might be to explore additional data/metrics that could be collected from within a project. Another way to improve the model is to collect data from additional teams. As previously mentioned, the data was collected from 80 project teams. While 80 projects might seem like a lot of projects, having 80 projects implies that the analysis was performed on the relatively small dataset of 80 rows of data. Thus, data could to be collected, across additional teams, which would likely help refine and improve the model.

Finally, this study was focused on student-based data science teams (i.e. graduate students taking applied data science class). The results might be different for other teams, such as software development teams or when reviewing projects with experienced industry professionals. Hence, additional studies could be explored within industry-based teams, across a range of domains.

6. Conclusion

In exploring the impact of the predictive algorithms, the baseline would predict that all teams were at risk, and so while all at risk teams would be identified, 46% of the teams would be treated as if they were at risk, even though they were not at risk. Thus, 100% of the teams “at risk” would be correctly identified. However, 0% of the teams “not at risk” would be correctly identified.

Using univariate feature selection approach with 4 features, the model provides a manager with opportunity to identify, on average 50% of the teams “at risk” (i.e. recall). At the same time, on average, 70% of the teams “not at risk” will be correctly classified as “not at risk”. Although the results reported are for a relatively small dataset, we believe that the characteristics of the classification model are stable because we used 1000 iterations of 5-cross-validation. In other words, the results are generalizable to new data, and if one applies the model to new teams, the class instructor or data science team manager would be able to detect approximately 50% teams at risk as well as correctly identify approximately 70% teams “not at risk”.

More generally, this study has several contributions. First, Kanban-based metrics were reviewed and additional metrics proposed. In addition, an analysis of the existing and proposed Kanban metrics was conducted. The analysis specifically explored each of the metrics predictive power with respect to team outcomes. Our results suggest that some of the newly proposed metrics were helpful in identifying teams at risk of poor performance (as compared to teams not at risk). Finally, as part of this work, several models were built to demonstrate that the selected metrics can help to identify teams at risk when the team is using a Kanban framework, with the best performing algorithm using KNN with univariate feature selection.

References

Ahmad

M. O.

Markkula

, & Oivo

(2013). Kanban in software development: A systematic literature review. In 2013 39th Euromicro Conference on Software Engineering and Advanced Applications, Santander, Spain, 2013, 9-16.

Ahmad

M. O.

Kuvaja

Oivo

, & Markkula

(2016). Transition of software maintenance teams from Scrum to Kanban. In 49th Hawaii International Conference on System Sciences (HICSS), 5427-5436.

Ahmad

M. O.

Dennehy

Conboy

, & Oivo

(2018). Kanban in software engineering: A systematic mapping study. The Journal of Systems and Software, 137,, 96-113.

Al-Aidaros

H. A.

, & Omar

(2017). Software project management approaches for monitoring work-in-progress: A review. Journal of Engineering and Applied Sciences, 12(15), 3851-3857.

Anderson

(2010). Kanban: Successful Evolutionary Change For Your Technology Business. Blue Hole Press, Sequim, Washington.

Baron

J.A.

(1994). Too bad it isn’t true. Medical Decision Making, 14(2), 107-107. doi: 10.1177/0272989X9401400202. PMID 8028462.

Benson

(2011) Kanban for Short Intense Projects: How We Used Kanban to Visualize Our Hiring Process Workflow and Make Our Lives Easier, Accessed July 18, 2019. http://personalkanban.com/pk/designpatterns/kanban-for-short-intense-projects-how-we-used-kanban-to-visualize-our-hiring-process-workflow-and-make-our-lives-easier/.

Berander

(2004). Using students as subjects in requirements prioritization. Proceedings, International Symposium Empirical Software Engineering 2004. ISESE’04, 167-176.

Bladt

, & Filbin

(2013). Know the Difference Between Your Data and Your Metrics https://hbr.org/2013/03/know-the-difference-between-yo

10.

Boyko

E. J.

(1994). Ruling out or ruling in disease with the most sensitive or specific diagnostic test: short cut or wrong turn? Medical Decision Making, 14(2), 175-179. doi:10.1177/0272989X9401400210. PMID 8028470.

11.

Catal

& Diri

(2009). A systematic review of software fault prediction studies. Expert Systems with Applications, 36(4), 7346-7354.

12.

Dvir

Lipovetsky

Shenhar

, & Tishler

(1998). In search of project classification: a non-universal approach to project success factors. Research Policy, 27, 915-935.

13.

Dvir

Lipovetsky

Shenhar

, & Tishler

(2003). What is really important for project success? A refined, multivariate, comprehensive analysis. International Journal of Management and Decision Making, 4(4), 382-404.

14.

El-Halees

(2008). Mining student data to analyze learning behavior: A case study, International Arab Conference of Information Technology (ACIT2008), Tunisia, 2008.

15.

Feitelson

D. G.

(Dec 2015). Using students as experimental subjects in software engineering research – a review and discussion of the evidence. ArXiv:1512.08409 [cs.SE].

16.

Fenton

N. E.

, & Pfleeger

S. L.

(1998). Software Metrics: A Rigorous and Practical Approach. 2nd edition, PWS Publishing Co. Boston, MA, USA.

17.

Hildebrandt

Gurevich

Stuke

Kastrup

, & Stuke

(2017). Neuropsychological testing and machine learning distinguish alzheimer’s disease from other causes for cognitive impairment. Frontiers in Aging Neuroscience. 9. doi: 10.3389/fnagi.2017.00114.

18.

Hong

Chen

, & Hu

(2009). Evaluation of the sensitivity and specificity of qualitative diagnostic test under normal assumption. Model Assisted Statistics and Applications. 4. doi: 10.3233/MAS-2009-0113.

19.

Höst

Regnell

, & Wohlin

(2000). Using Students as Subjects — A Comparative Study of Students and Professionals in Lead-Time Impact Assessment. Empirical Software Engineering, 5(3), 201-214.

20.

Hoff

K. J.

Lingner

Meinicke

, & Tech

(2009). Orphelia: predicting genes in metagenomic sequencing reads. Nucleic Acids Research, W101-105. 37.

21.

Hua

Xiong

Lowey

Suh

, & Dougherty

E. R.

(2005). Optimal number of features as a function of sample size for various classification rules. Bioinformatics, 21(8), 1509-1515. https://doi.org/10.1093/bioinformatics/bti171

22.

James

Witten

Hastie

, & Tibshirani

(2013). An Introduction to Statistical Learning with Applications in R.

23.

Kabakchieva

(2013). Predicting Student Performance by Using Data Mining Methods for Classification. Cybernetics and Information Technologies, 13, 1.

24.

Kahneman

, & Tversky

(1979). Intuitive prediction: biases and corrective procedures. TIMS Studies in Management Science 12, 313-327.

25.

Kirovska

, & Koceski

(2015). Usage of Kanban methodology at software development teams. Journal of Applied Economics and Business, 3(3), 25-34. ISSN 1857-8721.

26.

Kitchenham

B. A.

Pfleeger

S. L.

Pickard

L. M.

Jones

P. W.

Hoaglin

D. C.

Emam

K. E.

, & Rosenberg

(2002). Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8), 721-734.

27.

Kupiainen

Mantyla

M. V.

, & Itkonen

(2014). Why are industrial agile teams using metrics and how do they use them? in: Proceedings of the 5th International Workshop on Emerging Trends in Software Metrics. ACM, 2014, 23-29.

28.

Kupiainen

, Mäntyla M.V., & Itkonen J. (2015). Using metrics in agile and lean software development – a systematic literature review of industrial studies. Information and Software Technology, 62, 143-163.

29.

Lambert

(2014). Using Kanban in the recruitment process, The Social Tester. Accessed 18 July 2019. http://thesocialtester.co.uk/kanban-and-recruitment/

30.

Landry

, & Mc Daniel

(2015). Agile preparation within a traditional project management course, agile preparation within a traditional project management course, 2015 Proceedings of the EDSIG Conference, (2015).

31.

Lei

Ganjeizadeh

Jayachandran

P. K.

, & Ozcan

(2017). A statistical analysis of the effects of Scrum and Kanban on software development projects. Robotics and Computer-Integrated Manufacturing, 43, 59-67.

32.

Levesque

L. L.

Wilson

J. M.

, & Wholey

D. R.

(2001). Cognitive divergence and shared mental models in software development project teams. Journal of Organizational Behavior, 22(2), Special Issue: Shared Cognition, 135-144.

33.

Leybourn

(2013). Directing the Agile Organisation: A Lean Approach to Business Management. London: IT Governance Publishing: 160-66.

34.

Lipovetsky

Tishler

Dvir

, & Shenhar

(1997). The relative importance of project success dimensions. R&D Management, 27, 97-106.

35.

Mahnič

, & Zabkar

(2012). Measuring progress of scrum-based software projects. Electronics and Electrical Engineering, 18(8), 73-76.

36.

Mahnič

(2013). Applying Kanban Principles to Software Development, Proceedings of the IT for Practice 2013 16th International Conference on Information Technology for Practice.

37.

Middleton

, & Joyce

(2012). Lean Software Management: BBC Worldwide Case Study, Engineering Management. IEEE Transactions, 59(1), 20-32.

38.

Minaei-Bidgoli

Kashy

D. A.

Kortemeyer

, & Punch

W. F.

(2003). Predicting student performance: an application of data mining methods with an educational Web-based system. 33rd Annual Frontiers in Education, 2003. FIE 2003, Westminster, CO, 2003, pp. T2A-13 doi: 10.1109/FIE.2003.1263284

39.

Moore

Reff

Graham

, & Hackerson

(2007). Scrum at a Fortune 500 Manufacturing Company. Agile 2007 (AGILE 2007), Washington, DC, 2007, 175-180.

40.

Mujtaba

Feldt

, & Petersen

(2010). Waste and lead time reduction in a software product customization process with value stream maps. Proceedings of the Australian Software Engineering Conference, ASWEC, 139-148, Auckland.

41.

Nakazawa

& Tanaka

(2015). Prototype of Kanban Tool and Preliminary Evaluation of Visualizing Method for Task Assignment. In 2015 International Conference on Computer Application Technologies, Matsue, 2015, 48-49.

42.

Nakazawa

& Tanaka

(2016). Development and Application of Kanban Tool Visualizing the Work in Progress. In 2016 5th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), Kumamoto, 2016, 908-913.

43.

Ohno

(1988). Toyota Production System: beyond large-scale production. Productivity Press, 1988.

44.

Osmanbegović

, & Suljić

(2012). Data mining approach for predicting student performance. Economic Review – Journal of Economics and Business, X(1), May 2012.

45.

Ostergaard

(2016). Applying Kanban principles to electronic resource acquisitions with Trello. Journal of Electronic Resources Librarianship, 28(1), 48-52.

46.

Parikh

Mathai

Parikh

Sekhar

G. C.

, & Thomas

(2008). Understanding and using sensitivity, specificity and predictive values. Indian Journal of Ophthalmology, 2008 Jan-Feb; 56(1), 45-50.

47.

Pedregosa

Varoquaux

Gramfort

Michel

Thirion

Grisel

Blondel

Prettenhofer

Weiss

Dubourg

Vanderplas

Passos

Cournapeau

Perrot

, & Duchesnay

(2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, 2825-2830.

48.

Petersen

, & Wohlin

(2010). The effect of moving from a plan-driven to an incremental software development approach with agile practices: An industrial case study. Empirical Software Engineering, 15(6), 654-693.

49.

Petersen

, & Wohlin

(2011). Measuring the flow in lean software development. Software - Practice and Experience, 41(9), 975-996.

50.

Pewsner

Battaglia

Minder

Marx

Bucher

H. C.

, & Egger

(2004). Ruling a diagnosis in or out with “SpPIn” and “SnNOut”: a note of caution. BMJ, 329(7459), 209-213. doi:10.1136/bmj.329.7459.209. PMC 487735. PMID 15271832

51.

Project Management Institute (PMI). (2004). A Guide to the project management body of knowledge, third edition (PMBOK® guide). Newtown Square, PA: Project Management Institute.

52.

Power

(2014). Metrics for Understanding Flow, Agile Software Development Conference (Agile 2014), Orlando, FL, USA.

53.

Powers

(2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation. Journal of Machine Learning Technologies, 2(1), 37-63.

54.

Pulford

Kuntzmann-Combelles

, & Shirlaw

(1995). A quantitative approach to software management: The AMI handbook, Addison-Wesley Longman Publishing Co., Inc., 1995.

55.

Pressman

R. S.

, & Maxim

B. R.

(2015). Software Engineering: A Practitioner’s Approach, 8th Edition. McGraw-Hill, 2015.

56.

Runeson

(2003). Using students as experiment subjects – an analysis on graduate and freshmen student data. Proceedings of the 7th International Conference on Empirical Assessment & Evaluation in Software Engineering, 95-102.

57.

Salman

Misirli

A.T.

, & Juristo

(2015). Are students representatives of professionals in software engineering experiments? Proceedings of the 37th International Conference on Software Engineering, 1, 666-676.

58.

Saltz

(2015). The need for new processes, methodologies and tools to support big data teams and improve big data project effectiveness. in: Big Data (Big Data), 2015 IEEE International Conference on, Santa Clara, CA, USA, 2015, 2066-2071.

59.

Saltz

Shamshurin

, & Connors

(2017). Predicting data science sociotechnical execution challenges by categorizing data science projects, Journal of the Association for Information Science and Technology (JASIST).

60.

Saltz

Shamshurin

, & Crowston

(2017). Comparing Data Science Project Management Methodologies via a Controlled Experiment. In 2017 50th Hawaii International Conference on System Sciences, Waikoloa, HI, USA, 2017, 1013-1022.

61.

Scikit-learn.org. (2018). Univariate feature selection. https://scikit-learn.org/stable/modules/feature_selection.html.

62.

Scotland

(2009). Kanban, flow and cadence. Accessed Jul 24 2019. https://www.accu.org/content/conf2009/KarlScotland_kfc-development-accu2009.pdf.

63.

Sharma

, Yadav U.B., & Sharma

(2009). The concept of sensitivity and specificity in relationto two types of errors and its application in medical research. Journal of Reliability and Statistical Studies, 2(2), 53-8.

64.

Shen

, & Ju

(2007). On the measurement of agility in software process. Lecture Notes in Computer Science, 4470 LNCS, 25-36.

65.

Shenhar

Tishler

Dvir

Lipovetsky

, & Lechler

(2002). Refining the search for project success factors: a multivariate, typological approach. R&D Management, 32(2), 111-126.

66.

Sjøberg

D.I.K.

Anda

Arisholm

Dyba

Jorgensen

Karahasanovic

Koren

E. F.

, & Vokac

(2002). Conducting realistic experiments in software engineering. Proceedings International Symposium on Empirical Software Engineering, 1325.

67.

Sjøberg

D.I.K.

Hannay

J. E.

Hansen

Kampenes

V. B.

Karahasanovic

Liborg

N. K.

, & Rekdal

A. C.

(2005). A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9), 733-753.

68.

Sutherland

(2004). Agile development: lessons learned from the first Scrum, Cutter Agile Project Management Advisory Service. Executive Update, 5, 20.

69.

Sugimori

Kusunoki

Cho

, & Uchikawa

(1977). Toyota production system and kanban system materialization of just-in-time and respect-for-human system. International Journal of Production Research 15(6), 553-564.

70.

Svahnberg

Aurum

, & Wohlin

(2008). Using students as subjects – an empirical evaluation. In Proceedings of the Second ACMIEEE international symposium on Empirical software engineering and measurement, 288-290.

71.

Taibi

Lenarduzzi

Janes

Liukkunen

, & Ahmad

(2017). Comparing Requirements Decomposition Within the Scrum, Scrum with Kanban, XP, and Banana Development Processes. XP 2017: Agile Processes in Software Engineering and Extreme Programming, 68-83.

72.

Tichy

W. F.

(2000). Hints for reviewing empirical work in software engineering. Empirical Software Engineering 5(4), 309-312.

73.

Tishler

Dvir

Shenhar

, & Lipovetsky

(1996). Identifying critical success factors of defense development projects: a multivariate analysis. Technological Forecasting and Social Change, 51, 151-171.

74.

Trapa

, & Rao

(2006). T3 - tool for monitoring agile development,Proceedings - AGILE Conference, 2006, 243-248, Minneapolis, MN.

75.

Tudor

& Walter

(2006). Using an agile approach in a large, traditional organization. In Proceedings - AGILE Conference, 2006, 367-373, Minneapolis, MN.

76.

VersionOne Inc. (2018). The 12th annual state of agile report. Technical Report. VersionOne Inc. https://explore.versionone.com/state-of-agile/versionone-12th-annual-state-of-agile-report.

77.

VersionOne, Inc. (2016). The 10th annual state of agile survey. Technical Report. VersionOne Inc. http://www.agile247.pl/wp-content/uploads/2016/04/VersionOne-10th-Annual-State-of-Agile-Report.pdf.

78.

Wolf

Schroter

Damian

, & Nguyen

(2009). Predicting build failures using social network analysis on developer communication. In Proceedings of the 31st International Conference on Software Engineering (ICSE ‘09). IEEE Computer Society, Washington, DC, USA, 1-11. doi: 10.1109/ICSE.2009.5070503.

79.

Yerushalmy

(1947). Statistical problems in assessing methods of medical diagnosis with special reference to x-ray techniques. Public Health Reports, 62(2), 1432-39. doi: 10.1177/0272989X9401400202. PMID 8028462.

80.

Zhang

J. G.

(2011). Schedule management method study of middle and small software projects. in: 2nd IEEE International Conference on Artificial Intelligence, Management Science and Electronic Commerce, Dengleng, China, 1495-1498.

	All teams are “at risk”	All “not at risk”
Recall or sensitivity (true positive rate)	0.46	0 – undefined
Precision (positive predictive value)	1	0
Selectivity or specificity (true negative rate)	0 – undefined	0.54
Accuracy (% correct)	0.46	0.54
F1 (harmonic mean of precision and recall)	0.7	0 – undefined
Sensitivity and selectivity (harmonic mean)	0 – undefined	0 – undefined

A predictive model to identify Kanban teams at risk

Abstract

Keywords

1. Introduction

2. Background

2.1 Project management and project metrics

3.1 The use of students and their applicability to industry

3.2 Project context

3.3 Data collection

Table 1 Number of teams and students participated in the experiment

4.1 Linear regression

4.2 Comparison of predictive power of the metrics

4.3 Feature engineering/selection

Table 3 Performance of ML models depending on number of PCA components

6. Conclusion

References

Table 1
Number of teams and students participated in the experiment

Table 3
Performance of ML models depending on number of PCA components