The usability analysis of software loading tools in a commercial airline

The researchers coordinated with a commercial airline based in Saudi Arabia. The line maintenance managers of each station helped researchers ensure efficient communication with avionics technicians. All avionics technicians allowed their respective managers to give their contact details to the researchers. Then, line maintenance managers provided the complete list of the company’s avionics technicians alongside their contact information.

This study was approved by Mapua University Research Ethics Committees (FM-RC-21-75). Prior to the data collection, the researchers explained the study’s purpose to each participant. All participants willingly filled out the consent form. The questionnaire disseminated to participants’ company emails was created through Google Forms.

A total of 20 professional avionics technicians voluntarily participated in the study (Table 1). Although these technicians have varying base locations, their designated duties are considered the same level. All 20 participants were aircraft specialists from a commercial airline with 2 to 30 years of experience (mean: 15.65 years; SD: 8.62 years). However, their experience does not signify employee tenure in that specific commercial airline. All avionics technician-related experience was taken into consideration regardless of the company name. This approach is considered a good comparison because aviation companies operate software loading with different tools.

In this study, all participants were male. This coincided with the male-dominated occupation in the technician field, especially in commercial aviation [18]. They were also asked to estimate their performances when uploading the database using three tools. Although the completion time is self-reported and may have time differences in comparison to the actual performance, past studies supported the dissemination of a questionnaire in the absence of actual data [19, 20]. Furthermore, the line maintenance managers vouched for their technicians’ responses and skills. In the actual questionnaire, technicians inputted an estimated time of loading each tool and the researchers categorized them based on the recommended categories (≤60 minutes, 60 <  minutes <  90, ≥90 minutes). For instance, if one technician reported 58.5 minutes of floppy disk software loading completion, then it would be categorized as a response frequency number under ≤60 minutes. The same technique was followed through the rest of Table 1’s completion time. The average completion time was also calculated in the latter part of the discussion by computing the mean values based on the self-reported individual completion time for each software tool. Specifically, the sum of the self-reported individual completion time was divided by the total number of avionics technicians (20), resulting in the average completion time values for each corresponding tool.

SUS has been widely used since 1986 to investigate user experience aiming to improve human practices and develop technological aspects [17]. The three aircraft software loading tools were compared through usability analysis, specifically using the SUS scoring system. It was supplemented with completion time, which refers to the time that the technicians acquire to finish the software uploading task in the maintenance area.

Three sets of questionnaires, one for every software tool, were distributed among the respondents randomly (Table 2). Hence, the respondents answered a total of 30 questions. SUS questionnaire applies a 5-point Likert scale. The scale was converted from subjective features to figures to calculate the SUS score, whereas 1 point reflects strongly disagree and 5 points is pertinent to strongly agreed opinion. Then, the mean values from each observed factor (technician responses and question items) were computed. Next, the pointing system was adjusted because odd-numbered items were positive questions while even-numbered items were negative questions. SUS questions were arranged in an alternate positive-negative sequence to ensure that respondents fully understood each statement [21]. For positive questions (items 1, 3, 5, 7, and 9), the researchers subtracted one from the user response. For negative questions (items 2, 4, 6, 8, and 10), the researchers subtracted the user’s response from five. Next, the total adjusted SUS values for all items were calculated for each question. The sum was then multiplied by 2.5. The result yields numerical values from 0 to 100, where 0 is poor and 100 is excellent [21]. This scoring approach was applied to three software tools.

This study adopted the recommended SUS scales published by Bangor et al. [16]. Table 3 top table shows adjective ratings of the direct translation of SUS scores while the secondary table below identifies the acceptability range for the software aviation tool’s SUS scores. It was depicted that both scales must be used to interpret SUS scores accurately, but their gravity could be different depending on the context. For instance, an okay adjective rating could mean a satisfactory rating in a non-aviation context [16]. In this context, the acceptability range holds greater implications because aviation software loading is associated with people’s lives and business profit or loss.

The combination of ANOVA and Pearson correlation was deemed successful in comprehension and human behavior studies [22]. ANOVA tests the difference between means by comparing the variance between the groups about the variance within groups [22]. It determined the significant difference among the means of SUS score, experience, and completion time with three aviation software tools. It was coupled with Tukey’s HSD test to identify the specific tools affected by the significant variables. On the other hand, the Pearson correlation coefficient analyzes the relationships among the obtained variables [22]. This statistical technique checks the direction, strength, and hypotheses among the variables through the covariance approach [22]. The study’s primary variables were the three aviation software loading tools, SUS scores, years of experience, and completion time. These statistical techniques were performed using SPSS 26.

3.1 Descriptive statistics

A total of 15 technicians perceived they could complete the usual aircraft software update powered by floppy disks in more than 90 minutes, and only 5 technicians could perform it faster. Teledyne PMAT produced an almost similar result to floppy disks. More than half of the participants (12 out of 20) perceived they could complete the software loading for at least 90 minutes, and 8 avionics technicians could perform their duties for less than 90 minutes. Meanwhile, there was a huge difference when the completion time of floppy disks and Teledyne PMAT were compared to MBS mini PDL. Interestingly, 16 out of 20 avionics technicians perceived their work to be faster using MBS mini PDL, specifically at most 60 minutes. Only 4 participants could consume more than 60 minutes.

3.2 SUS scoring for floppy disk, Teledyne PMAT, and MBS mini PDL

Table 4 shows the years of experience and SUS score of each technician using floppy disks, Teledyne PMAT, and MBS mini PDL. It was arranged in highest to lowest years of experience, which reflected that SUS scores did not correlate with SUS scores across all three tools. On one hand, technicians’ SUS scores vary for each software loading tool.

For floppy disk SUS scores, avionics technicians gained SUS scores ranging from 7.5 to 57.5. Unfortunately, these scores did not meet the required 70 SUS cut-off score [16]. Four avionics technicians scored low marginal (50 to 57.5) and 16 avionics technicians had poor scores (7.5 to 47.5). Low marginal groups have the potential for improvement while poor score groups are considered unacceptable [16]. Technician No. 17, with an experience of 9 years, had the lowest SUS score (7.5) among the respondents. Meanwhile, Technician No. 7 had the highest score of 57.5 for using floppy disks for 18 years. Table 5 shows the mean responses for all questions, items’ respective deviations, and computed SUS scores. Considering 70 SUS score cutoff [16], floppy disk did not meet the required cut-off as it scored 34.63 out of 100. Meanwhile, technicians’ SUS scores using Teledyne PMAT became more diverse compared to floppy disks. Specifically, their SUS scores ranged from 7.5 to 90. 13 avionics technicians produced unacceptable SUS scores (<50). Technicians no. 17 and 19, with experiences of almost 10 years, had the lowest SUS score (7.5). On the other hand, two technicians (Nos. 4 and 9) were categorized as low marginal and other technicians (Nos. 10 and 20) were tagged as high marginal. Furthermore, three technicians (Nos. 8, 11, and 14) yielded acceptable SUS scores. Technician no. 14 who has 15 years of work experience provided the highest SUS score (90), followed by technician no. 8 (80) with 14 years of aviation software experience, and technician no. 11 (70) with 4 years of aviation software experience. Table 5 shows the computation details of SUS scores for the questions about Teledyne PMAT. The Teledyne PMAT’s SUS score was comparatively higher than floppy disks. This aviation software tool scored 39.38 out of 100, which was still considered inadequate [16]. In addition, SUS scores of each technician using MBS mini PDL were relatively high compared to the other two options. There was only one avionics technician (No. 1) with 28 years of work experience who produced an unacceptable score (SUS = 40). Technician No. 1 had the lowest SUS score among all technicians who used MBS mini PDL. Another avionics technician (No. 9) was categorized as high marginal (SUS = 67.5). Moreover, 18 avionics technicians generated at least 70 SUS scores. Technician No. 14 who has an experience of 15 years had the highest SUS score of 100. Following Table 5, the final SUS score using MBS mini PDL was calculated. This aviation tool scored 78.5 out of 100, the highest among the three aviation tools with a huge margin. If translated to the adjective rating, the SUS score was considered good [16].

Based on the generated results, the null hypothesis was rejected for SUS (p-value: 0.001) and completion time (p-value: 0.001) and accepted for experience (p-value: 1.000). Both SUS and completion time had a statistically significant difference between means contrary to experience. Tukey’s HSD identifies significant differences among specific groups encompassed within the dependent variables, which ANOVA test failed to identify [22]. Table 6 provides Tukey’s HSD results. In this context, specific groups pertain to the three aviation software tools. Tool 1.00 refers to floppy disks, tool 2.00 is Teledyne PMAT, and tool 3.00 denotes MBS mini PDL. Considering SUS scores and completion time, the association between tools 1 & 3 and tools 2 & 3 presented statistical differences. However, there were no significant differences between tools 1 & 2. Findings under experience could be disregarded since this variable did not yield significant statistical relationships.

In this subsection, the significant linear relationship among three tools, SUS scores, years of experience, and completion time in minutes was elaborated. Parallel to ANOVA and Tukey’s HSD tests, experience did not generate any significant differences when compared to other variables. Meanwhile, the tool and SUS were positively and significantly correlated (r = 0.704, p <  0.01). The magnitude of the association is approximately strong (0.5< |r|). The aviation software tool also had a significant linear relationship with completion time (r = -0.651, p <  0.01). The direction of their relationship was negative, which posited an inverse relationship between the tool and completion time. Their relationship’s magnitude was strong in a negative direction (–0.5< |r|). Similar to the preceding direction and magnitude, SUS and completion time also had a significant negative relationship (r = -0.619, p <  0.01).

4.1 Evaluation of aviation software tools and their corresponding variables

Floppy disk’s SUS score (34.63) and average completion time (99.5 minutes) garnered a poor adjective rating and unacceptable range. Thus, avionics technicians should not load aircraft systems through floppy disks. The usage of floppy disks was deemed the most inefficient tool in this study, and thus must no longer be used by any aviation companies. Another software tool was compared to a floppy disk and researchers concluded that the compared tool yielded faster data transfer rates than a floppy disk; however, a floppy disk was deemed more user-friendly [8]. However, a user-friendly tool did not entail productivity in the present study. Avionics technicians’ SUS scores exhibited sentiments of being burdened, uneasy, and impractical when performing tasks using a floppy disk. While floppy disk is a convenient data storage device, inventors prefer improving technology software tools [7]. As technology constantly evolves, there is a need to stop patronizing old software tools, such as floppy disks, and start maximizing the functions of modernized aircraft tools. Overall, avionics technicians’ SUS scores using floppy disks had a poor adjective rating and an unacceptable range. The coordinated results for both SUS scales supported the outdated features of floppy disks. While some data would be difficult to merge between floppy disks and the other two tools, researchers suggested to invest in software architect with expertise in old and new technologies.

Teledyne PMAT’s adjective rating was considered okay but the acceptability range was deemed unacceptable because the SUS score was 39.38 and and average completion time was 86 minutes. Inconclusive results were normal when two scales were compared due to range differences. The current study considered a greater position on the acceptability range more than the adjective rating because aviation’s navigation and software loading systems hold critical facets of humanity and business [16, 17]. Similar to the current study’s arguments, a past study also considered floppy disks as more inconvenient than Teledyne PMAT [2]. Since Teledyne PMAT has multiple side functions, these added features cause avionics technicians challenges in navigating the tool. Some of its side functions include managing corrective actions and scheduling preventive maintenance. However, these are not essential in loading the navigation and software loading system. Therefore, Teledyne PMAT was considered a complex tool with a sloppy user interface. This tool was not recommended by researchers. Nevertheless, Teledyne PMAT was more convenient due to its ability to connect with an internet connection. This function helped avionics technicians maneuver the aircraft’s safety and mitigate data threats. While Teledyne PMAT has advantages, a study mentioned that it could take 4.5 hours to load an aircraft information management system [9]. The aforementioned system was entirely different from the observed software loading system in the current study, and this instance supported the novelty of the current study’s findings.

MBS mini PDL was found the most convenient and efficient tool. Specifically, it resulted in 78.5 SUS score and 58.3 minutes of average completion time. These findings showed that MBS mini PDL was an excellent tool with acceptable values. This aviation software tool should be prioritized in all aviation systems to reduce non-value-added activities in preparing aircraft. MBS mini PDL exceeded floppy disks in terms of storage and cloud market [7], which supported the current study’s findings. MBS mini PDL was confirmed to have a higher capacity and be more reliable. Although PDL has an alternate tool, such as onboard data loaders, PDL was more convenient because it could be connected to external networks and devices [5]. This function was deemed necessary in every aircraft because of the continuous database version updates. Another study supported the value of PDL’s data transmission as the tool automatically connects navigation systems with airline company protocols and other needed software entities [1]. A study noted that employees who valued software usability generated acceptable SUS scores [23]. Hence, this past and current study upheld the importance of using an appropriate software tool to maximize employees’ productivity more than other variables.

It was also deemed noteworthy when professionals utilized a technological information system and concluded that there were significant differences between SUS scores and completion time [24]. The observed technological tool passed the usability test through the following features: customization, comprehensibility, automation, and system capacity. In this study, MBS mini PDL displayed a user-friendly interface, multipurpose functions, and large capacity. Therefore, avionics technicians produced better performance in uploading the aircraft’s database system using MBS mini PDL.

Furthermore, findings demonstrated the increased SUS scores and decreased completion time while comparing floppy disks & MBS mini PDL and Teledyne PMAT & MBS mini PDL. MBS mini PDL was the common ground among all comparisons, which supported the significance of its corresponding variables. Meanwhile, a sole analogy between floppy disks’ and Teledyne PMAT’s SUS scores and completion time was found insufficient due to the insignificant proximity of values. Both SUS scores and completion time did not involve any gigantic improvements or deteriorations despite multiple comparisons.

Moreover, the researchers found that veterans’ experience did not have any advantages over novice avionics technicians. In a normal work setting, it was a common misconception that years of experience would make employees faster workers but the current findings provided evidence that they lacked connection. Contrastingly, a past study revealed that users with more extensive experience returned better SUS scores compared to those with limited experience [25]. Although both studies analyzed a technology tool, result differences occurred due to the user’s nature. The past study focused on common citizens who utilized certain websites, while the current study analyzed employees’ set-up tools. Thereby, unique workplace arrangements had different implications compared to daily technologies. Another study produced contrasting results, where project managers’ and computer programmers’ experiences contributed positively to workplace duties [23]. While the past study assessed workplace scenarios, they failed to specify job scopes and software tools. The current study discussed avionics technicians’ responsibilities and aircraft tools, which enlightened SUS findings. The researchers concluded that avionics technicians’ tenure was irrelevant to the tool’s perceived usefulness and work performance.

4.2 Managerial implications

Since MBS mini PDL was less time-consuming, avionics technicians and other aircraft employees could utilize the preserved 27.7 minutes to 41.2 minutes for other aircraft preparation. These minutes were calculated by subtracting the average completion time of MBS mini PDL from Teledyne PMAT and floppy disks. One contribution was pertinent to the optimization of flight schedules. Aircraft could take off or return earlier than the scheduled time, and several round trips could be added to the schedule. Considering that the longest amount of time that could be saved was 41.2 minutes, this could contribute to an additional 34 aircraft schedules (depending on the route) in one day cycle. A total of 34 aircraft were calculated by dividing the total minutes in one day (1440) by the saved time (41.2), which resulted in 34.95. Aircraft couldn’t be interpreted in decimal, hence, researchers used round down to the whole number approach considering that round-up would mean inadequacy. Since the observed airline company has 30 aircraft only, this study would utilize 30 aircraft instead of 34 aircraft. If additional routes were added to each aircraft, the company would increase its sales by receiving additional flight bookings. For instance, the researchers used a conservative approach of 160 booked passengers daily in a commercial airline [26]. The passenger capacity was based on Boeing and Airbus as they are the aircraft models that frequently used the three software loading tools evaluated in the study. Local flight, one-way trip, economy class, and the lowest rate was considered using the Skyscanner application. According to Skyscanner, approximately 114 USD is the cheapest local flight as of December 2022.

Following the succeeding formula, aircraft could increase its yearly sales using Equation 1:

30 aircraft × ( 160 passengers / aircraft ) × ( 114 USD / passenger ) × ( 365 days / year ) = 199 , 728 , 000 USD / year(1)

Most importantly, a decrease in airline maintenance’s completion time is equivalent to savings. For example, the researchers performed a cost-saving analysis by comparing the performance of floppy disks and MBS mini PDL. These two tools were compared because floppy disks yielded the weakest performance while MBS mini PDL received the most acceptable rating. In the observed commercial airline, the avionics technician’s labor rate is approximately 50 USD [27]. This company has 30 aircraft and the software loading task is performed 13 times a year.

The following calculation shows the annual salary of one avionics technician by performing aircraft software loading using floppy disks (Equation 2):

99.5 mins × ( 1 hr / 60 mins ) × ( 50 USD / hr ) × 30 aircraft × ( 13 repetition / year ) = 32 , 337.5 USD / year(2)

On the other hand, this computation represents the avionics technician’s annual salary using MBS mini PDL (Equation 3):

58.3 mins × ( 1 hr / 60 mins ) × ( 50 USD / hr ) × 30 aircraft × ( 13 repetition / year ) = 18 , 947.5 USD / year(3)

These two equations disregarded the possibility of having two or more avionics technicians in one aircraft. Thus, researchers applied a conservative approach to the potential savings’ analyzation. By subtracting Equations 3 from 2, the airline company could save 13, 390 USD every year if avionics technicians would utilize MBS mini PDL to update and load the aircraft’s database. This analysis supported that MBS mini PDL was cost-effective, efficient, and dependable.

At times, aircraft may land at a different location and software update must be performed. However, aircraft on ground at an overseas airport cost an estimated 30 USD every minute [6]. The rough estimate may vary depending on the airport location and aircraft type, though. Nevertheless, the researchers considered the declared cost from the study of Milne & Kelly to gauge the probable estimated savings [6]. Through an equally conservative approach, an estimated landing at ONE different airport was utilized for all aircraft.

If floppy disks were utilized in loading the aircraft database at an overseas airport, the following computation should be followed (Equation 4):

99.5 mins × ( 30 USD / min ) × 30 aircraft × ( 1 repetition / year ) = 89 , 550 USD / year(4)

Meanwhile, MBS mini PDL would incur an overseas aircraft on ground cost using Equation 5:

58.3 mins × ( 30 USD / min ) × 30 aircraft × ( 1 repetition / year ) = 52 , 470 USD / year(5)

Therefore, the commercial airline must utilize MBS mini PDL to save $37,080 each year following amount for the aircraft on ground costs. The total on ground saving of $37,080 was calculated by subtracting equation 5 from equation 4 because it was deemed the on ground cost derived from the poorest tool (floppy disk) in comparison to the best tool (MBS mini PDL). If companies were to benchmark these corresponding costs, they could save an amount that could be added to the cost-benefit analysis.

Table 7 reflects the summarized benefits of utilizing MBS mini PDL and sample cost comparison between floppy disks and MBS mini PDL. The cost-benefit ratio was not computed since not all aviation costs were determined in the study. Nevertheless, the findings presented an enormous increase in sales amounting to 199,728,000 and a total annual savings of 50,470 USD for both labor rate and overseas aircraft on ground. Hence, the commercial airline could save 199,778,470 USD annually if avionics technicians would utilize MBS mini PDL for the aircraft’s software loading system. This immense amount could be utilized for process improvements, employee engagement, passenger discounts, and other airline expenses.

Apart from the presented savings, researchers suggested that avionics technicians must have undergone a rigorous training process to ensure that associated risks with MBS mini PDL are eliminated. Aviation companies are encouraged to perform a series of MBS Mini PDL training. Licensed trainers and tenured technicians should spearhead the training program. Quarterly refresher courses should be given in a period of one to two weeks. On top of this, technicians must receive 100% scores on theoretical and practical examinations after refresher courses. They should only be given a maximum of two tries to take both examinations. If they fail, their managers ought not to deploy them to aircraft with MBS mini PDL loading system. Instead, these technicians must attend additional MBS mini PDL training and cross-train with other aircraft functions to keep themselves productive. Overall, this suggestion would ensure that human lives aren’t jeopardized once MBS mini PDL is integrated into the aircraft.

The researchers presented a comprehensive assessment of three aircraft software loading tools using SUS and statistical techniques. However, they also acknowledged the study’s limitations. First, the study used the perceived or self-reported completion time based on avionics technicians’ responses. This occurred due to aircraft scheduling and tool availability restrictions in the airline. The actual use of technological tools was mostly preferred by participants to keep their engagement in the study [28]. Hence, future researchers were advised to conduct an actual time study to compare perceived and actual completion time. It should be noted that months of preparation should be arranged beforehand since most airlines require several protocols. Nevertheless, past studies disclosed that perceived data is acceptable if it is approved by a third-party individual [19, 20]. In this study, avionics technicians’ responses were verified and approved by their respective line maintenance managers, which insinuated the completion time’s reliability.

Second, the study focused on one commercial airline. Airlines have different aircraft types and their technicians can employ varying software techniques. A past study also noted that proficiency in software tools could be associated with an organization’s values and the country’s culture [29]. Thus, future researchers could evaluate other airline companies, including local and global scales. This approach could help aircraft stakeholders get a wide array of visions on software tools’ usability. Still, the current study’s framework was deemed to have a solid foundation since the usability of floppy disks, Teledyne PMAT, and MBS mini PDL were evaluated through SUS with corresponding statistical methods.

Third, the researchers employed a conservative approach in the cost-benefit analysis. Future researchers could compare both conservative and aggressive approaches. An aggressive approach would entail separate forecasting of sales, demand, and expenses. This limitation stemmed from the SUS questionnaire itself. While SUS provided noticeable benefits for technicians and aviation companies, its subjective approach was deemed inadequate in covering all areas of technicians’ capabilities and aviation software tools. To cover this area, interested researchers must apply machine learning algorithms [30]. These AI-powered algorithms could interpret in-depth data more comprehensively. But then, the current findings were sufficient because the conservative approach already supported the huge benefits of utilizing MBS mini PDL over other software tools.