Embedding Secure Coding Instruction Into the IDE: Complementing Early and Intermediate CS Courses With ESIDE

Contributions

We are exploring methods for overcoming some of these barriers to integrating secure coding instructions across a curriculum. Our approach is to utilize the programming environment of the student and to deliver real-time secure programming instructional support as students write code. In exploring support outside of lectures and other classroom activities, we aim to minimize impact on instructors and existing course goals and learning materials. Instead, our goal is to complement time spent in the classroom by adding instructional support in secure programming while students are working on their coding skills through labs and assignments. We have prototyped this approach with an Eclipse plug-in called Educational Security in the Integrated Development Environment (ESIDE; Whitney, Lipford, & Chu, 2015; Zhu, Lipford, & Chu, 2013a). ESIDE provides the following capabilities:

Track students’ code writing process in real time for instances of vulnerable code.

In our initial development and research with ESIDE, we targeted advanced computing students as they introduced real-world vulnerabilities into web-based applications. In multiple evaluations, we demonstrated that ESIDE can improve the security awareness and knowledge of these students (Whitney, Lipford, & Chu, 2015; Zhu, Lipford, & Chu, 2013a). In this article, we investigate how to support beginning computing students, who have minimal programming and security knowledge. We adapted ESIDE for these students and the kinds of code constructs and security issues contained within. We report on the results of three studies with students in three courses over 2 years. Our results indicate that ESIDE can be utilized by early computing students, leading to increased awareness and knowledge of secure programming. We also highlight the challenges of scaffolding and timing of the secure coding instruction to increase learning and skill development.

Review of Literature

A variety of industry and government organizations have identified secure coding principles and knowledge (CERT, 2017; OWASP Foundation, 2016a) and promoted the integration of those practices into computing curricula. A course on secure programming is commonly offered as an elective later in a student’s education. Yet, across computing programs, there is little uniformity on course topics or prerequisites (Perrone, Aburdene, & Meng, 2005). Additionally, many researchers believe an elective course late in a student’s progression is too little, too late (Bishop & Frincke, 2005; Burley & Bishop, 2011; Taylor, Hochheiser, Azadegan, & Leary, 2009). Only a subset of students may be willing or able to enroll in the course. This leaves many students who never receive secure coding instruction. Additionally, by the time students enroll in the course, they have already developed a set of insecure programming practices that need to be adjusted. Even students specializing in security may only receive minimal training on secure coding practices. For example, in the curricular guidelines for the National Center of Academic Excellence in Information Assurance Education (National Security Agency Central Security Service, 2016), there is only one optional knowledge unit on secure programming practices out of 17 required and 47 optional knowledge units (National IA Education & Training Programs, 2016).

Researchers at Towson University have explored integrating secure programming instruction into broader computing curriculums, proposing “security injections” checklists with accompanying exercises that cover secure programming concepts that can be used in various courses, starting in the first programming course. These “security injections” are targeted lab modules (e.g., integer overflow, buffer overflow, input validation) that contain level-dependent secure coding tasks tailored for CS0, CS1, and CS2, checklists based on security principles (completed by the student), and scorecards (standard means for evaluation). The researchers tested the viability of their approach at multiple institutions and discovered an increase in security knowledge and a positive reception to the security topics (Taylor et al., 2009; Taylor & Kaza, 2011).

While security injections introduce secure coding concepts to students, they still require substantial time spent in class. In addition, they do not provide students with practice in applying those skills on their own code. Security clinics attempt to remedy this. Inspired by the concept of writing clinics, security clinics involve a teaching assistant or tutor meeting with a student one-on-one to go over specific pieces of code the student is working on. The tutor provides feedback on the security issues in that code and how to improve the programming practices.

These successful approaches demonstrate that introducing secure programming is possible, even for a beginning computing student. However, classroom time devoted to such content within programming courses will always remain limited. We are exploring a complementary approach, to support students outside of the classroom, as they are practicing and forming their coding habits through programming assignments and labs. Thus, we are adding secure coding instructional resources into the IDE, where students can be supported throughout their entire academic progression.

IDE Integration: Theory and Influential Practices

ESIDE was inspired by integrative learning, which is a process of making connections among knowledge and experience acquired across curricula. The process includes the blending and synthesis of multiple perspectives which ultimately result in an outcome that is greater than the sum of its parts (Brown Leonard, 2012; Huber & Hutchings, 2004). According to Brown Leonard (2007), “the emergent theory is that integrative learning is a developmental process that can be cultivated by an intentionally designed academic program” (p. 208). In addition, Caine and Caine (1991) describe how the brain continuously searches for common patterns and connections and that these patterns “tie together bits and pieces of information and give cohesive meaning and purpose to our daily learning” (p. 111).

Our goal for ESIDE is to enhance the learning process through the facilitation of connections between classroom principles and secure coding practices while the student is “in the moment.” We accomplish this by delivering contextually relevant, real-time educational materials in the student’s IDE. Although little research has been done on modifying IDE’s to teach nonfunctional requirements such as security, entire IDE’s have been developed and used as educational tools for teaching programming (Kölling, 2010; Paterson, Cheng, & Haddow, 2009; Ponzanelli, Bavota, Di Penta, Oliveto, & Lanza, 2014). These efforts provide some evidence that the IDE can be used as a learning environment.

ESIDE: Educational Security in the IDE

ESIDE was conceived after initial development of a tool for professional developers named Application Security in the IDE (ASIDE). ASIDE’s purpose is to provide interactive vulnerability detection and remediation assistance for developers, improving their awareness, knowledge, and practice of secure programming (Xie, Chu, Lipford, & Melton, 2011; Zhu, Xie, Lipford, & Chu, 2013b). Our first ESIDE implementation provided educational interventions for advanced students (in our case a senior and master’s-level web development course) as they become exposed to real-world coding vulnerabilities. In this article, we describe the version of ESIDE targeted toward beginner and intermediate students.

Our current prototype was developed for the Eclipse IDE for Java, and works by continuously monitoring the student’s code writing process for instances of patterns that match a predefined set of targeted code. When ESIDE finds a match, it initiates a layered educational intervention based on the code the student is writing. In the following example, the student is attempting to obtain input from a user with JOptionPane.showInputDialog(null, popUpTitle, message). In doing so, ESIDE places a discrete warning icon (Figure 1) in the left margin of the code editor. Hovering over the icon reveals a short message that encourages further interaction (Figure 2). When the student clicks the icon, ESIDE generates a context-specific list of options (i.e., input validation for filtering string input), with a short explanation alongside each option (Figure 3). ESIDE also provides an option to access an additional page that provides a more comprehensive explanation of what was discovered, why it is important, and how to integrate the provided principles into coding practices (Figure 4). The pages are expandable, showing high-level explanations that should take a typical student about one minute to review. From there, students can drill down into topics and examples that are relevant to the code they are working on. OPEN IN VIEWER

OPEN IN VIEWER

Figure 2.

Hover warning.

OPEN IN VIEWER

Figure 3.

Input validation remediation choices.

OPEN IN VIEWER

Figure 4.

Input validation support page.

ESIDE can also auto-generate remediation code for some types of vulnerabilities (Figure 5); in this case, the student filtered a string with ESIDE’s getValidSafeString method. When code is auto-generated, educational information is also included as comments. We utilized the auto-generated code for intermediate computing students, but not for beginning students, as we discuss later. OPEN IN VIEWER

Early Student Learning Concepts and ESIDE Content

As the student moves through the instructional process, there are key points where secure coding concepts and practices can complement a course’s instructional content. The first point occurs with early students as they are learning the basic principles of programming. We identified the following concepts by reviewing syllabi, courses, and faculty/government/industry recommendations (CERT, 2017; OWASP Foundation, 2015b; SAFECode, 2015; Towson University, 2016). While the following draws from the typical Java learning progression, these interventions are also applicable to other languages.

Early students develop a foundation of coding knowledge by mastering the principles of variables, objects, assignment statements, conditional statements, loops, i/o, arrays, and functions. Secure coding lessons should complement this process in two stages. The first stage is during the period where students control all their data and have no external data entering their code. At this point, it is important for them to learn the following:

Data types: Students should know the limits/ranges of each data type (e.g., byte = −128 to +127, short = −3,768 to + 3,767) and the problems that could arise if the wrong data type is used (e.g., expecting a number and using a letter).

The commencement of the second stage begins with the introduction of external data into the student’s code (e.g., console input, JOptionPane). At this point, it is critical that students begin to develop awareness for security issues by reflecting on what might happen to their program if it receives unexpected/untrusted input (e.g., strings instead of numbers). Once they have considered possible problems, they should learn how to validate obtained data. The following is a list of input validation points that students should learn how to implement at this stage:

Data type checks: Know how to verify that numbers are numeric characters and names are alphanumeric.

With consideration to these learning points, we developed ESIDE to initiate educational interventions by searching for code patterns they would typically use during this stage, based on the assignments and code utilized in the courses we evaluated. The following is a list of the code patterns ESIDE searched for:

Scanner Class: next, nextLine, nextInt, nextDouble, nextLong, nextShort, nextFloat, nextBigDecimal, nextBigInteger

We then aligned the security concepts directly to the written code to facilitate the connection between classroom principles and secure coding practices. For example, students were asked to obtain user input with JOptionPane as part of their lab. When students wrote the code necessary to accomplish the coding task, ESIDE intervened with a message explaining the importance of input validation (Figure 3). By doing so, ESIDE provided students with the opportunity to directly connect their coding practices with potential security problems.

Intermediate Student Learning Concepts and ESIDE Content

Students at the intermediate level expand their knowledge base by learning how to apply such things as exception handling, recursion, unit testing, application programming interfaces, hash maps, and simple graphical user interfaces. During this process, students should continue to consider potential problems with unexpected/untrusted input while they learn how to apply the following additional security concepts:

Input validation: Check that the data are of the correct type, in a reasonable range, and of reasonable length.

To ensure intermediate students were exposed to both these and the early learning points, we built upon the earlier version and included the following additional code patterns for the intermediate version of ESIDE.

File Class: File

The main goal of integrating these security concepts into the intermediate student’s learning process is to enhance their security awareness. ESIDE accomplishes this by providing data sanitization concepts, which go into greater depth than those available to early students. Thus, ESIDE’s instructional pages for the intermediate students provided educational support for three areas: InputValidation, OutputEncoding, and DirectoryTraversal.

Intermediate students also learn how to create and interact with various classes. To facilitate this development in a secure manner, we designed ESIDE to provide code generation with instructional comments. The purpose is to help the student begin to understand the process of implementing classes that are specifically written for the purpose of security such as the OWASP Enterprise Security application programming interface (OWASP Foundation, 2015a).

The most important goal for all stages is that students identify potential vulnerable areas in their code and use problem-solving skills to remediate security issues. By doing so, they are on their way to becoming robust programmers.

Previous ESIDE Research

Our initial version of ESIDE was similar to the professional ASIDE prototype, targeting senior undergraduate and master’s students in an advanced web development course, introducing real security vulnerabilities in web application assignments. This version was evaluated in both laboratory and field studies. ESIDE helped students become cognizant of their coding behavior and led to an increase in both security awareness and secure programming knowledge (Whitney, Lipford, & Chu, 2015; Zhu, Lipford, & Chu, 2013a). This article evaluates whether this technique is appropriate for a very different set of students—those just learning general programming. We next describe our formative evaluations of ESIDE, as well as an empirical study investigating the incentives for using ESIDE.

Methodology

We examined ESIDE in one early and one intermediate Java course at Elon University in North Carolina. This site was chosen as our university does not use Java or Eclipse in early courses. The activities that the students worked on during the studies were part of the class, but participation in the studies was voluntary. We provide an aggregate view of these participants in Table 1.

OPEN IN VIEWER

Table 1.

Study Summaries.

School/class	Level	Participants	Female/male	Study type
Elon CS1	Early	61	25/36	Lab assignment
Elon CS2	Intermediate	22	9/13	Lab assignment

The methodology for each study followed a similar format. The studies were conducted during a 90-minute lab session—three sections for the CS1 course and one section of CS2. We started with a study overview, had students sign consent forms, asked students to install ESIDE into their IDE, and then explained basic ESIDE use to the students. The students then worked on a programming assignment for approximately 60 minutes, interacting occasionally with ESIDE when a warning appeared in their code. Discussions were facilitated regarding ESIDE impressions. Following completion, ESIDE automatically collected the written code and interaction logs. Students also completed demographic surveys, and a pre (CS1) and post (all) test of secure programming knowledge. Knowledge surveys contained multiple-choice questions focused on awareness of security concepts related to software as well as specific mitigations for security issues.

With the instructors, we developed a coding assignment to be used for each lab, reflecting course content. The course did not include any other secure coding instruction. CS1 students worked on a grade averaging assignment which required them to use the Scanner class to collect a username, amount of grades, individual grades, place them into appropriate variables, and then develop methods to calculate the average. CS2 students worked on a file access activity, writing a program that would compare user-provided data against two csv files. Both the CS1 and CS2 assignments provided the opportunity for students to implement classroom lessons while also providing us the opportunity to complement the classroom lessons with secure coding support.

Observational Data

We logged when ESIDE was running, the amount and type of vulnerabilities found, and students’ interactions with ESIDE (e.g., clicking on the ESIDE icon). The study facilitator took notes of observations, discussions, and comments during and following each session. We also noted faculty perceptions, when they were present. In addition, we included open ended questions as part of the postsurvey to obtain student impressions and perceptions of ESIDE. Open coding was performed by the primary author on the notes from the sessions and survey responses, looking for common responses and patterns of behavior. Any student quotes provided later are taken from the surveys.

CS1 Study

For the CS1 study, 61 students consented to participate in the study (e.g., discussion, observation, coding logs, etc.) Of 61, 42 completed our pre- and posttest assessments in a counterbalanced fashion. At the end of the study, ESIDE submitted 33 interaction and coding logs to a central location. The number of logs submitted was less than the total number of participants due to a Citrix configuration issue in the lab, which was not discovered until after the study. Based on the interaction logs, ESIDE provided an average of five code warning icons per student during this session. Students initiated interaction with these warnings with an average of 5.15 times. All of the 33 students from whom we received logs interacted with the first two educational layers (icon and popup) and 15 accessed the third educational layer (instructional page). Twenty-two of the students implemented ESIDE support into their code in a manner such as the following:

String nextLine = ValidateInput.getValidAlphaNumeric(keyboard.nextLine(), keyboard);

During the first of three sections, classroom observations revealed minimal interaction with ESIDE. The instructor explained that this behavior was appropriate and expected due to the complexity of the Eclipse IDE. The instructor further described that Eclipse warnings easily confuse the early students and that the provided explanations are too advanced. The instructor had thus previously advised students to ignore Eclipse warnings and to figure out why they were receiving a warning based on the materials they had learned in class. For the other two sections, we then informed students that ESIDE’s content was written to their level. Doing so appeared to relieve aversion to interacting with any Eclipse warnings, and led to more interaction: “better explanation as to what was wrong with code than messages given by eclipse” (participant no. 77) and “the pop-up windows were helpful and wording was understandable” (participant no. 53).

Those who chose to implement ESIDE’s code recommendations (22 of the 33 participants) appeared to understand why it was important to do so and how the remediations would make their code more secure: “it helped me understand what I was doing wrong” (participant no. 58).

However, a small group of students were not prepared for the lab and needed to learn the concepts in the first place rather than practicing them. These students spent time searching ESIDE for the meaning of classroom content, which was not beneficial for them.

CS2 Study

In total, 22 participated and 21 completed the posttest assessment. At the end of the lab, ESIDE was unable to send interaction logs due to another unforeseen conflict with the Citrix lab configurations, which was only discovered after the study. However, web server logs revealed that 11 students navigated to the third educational layer (instructional page).

During this study, ESIDE introduced the concept of path traversal as related to accessing files during the lab assignment. Overall, students had difficulty understanding file paths as evident by their many questions/difficulties with providing the correct path to the files they were trying to access. When asked how many use or have used paths to navigate their computers, only a couple admitted to doing so. Thus, many graphical user interface-driven students had trouble understanding path traversal, and thus the security issues that could arise.

When ESIDE provided a warning icon, students reported that they felt something was wrong with their code. A common topic in response to the icon was whether ESIDE could help make their code “work.” This opened the discussion on the concept of secure/robust versus functional code. Many at this level had not considered that just as there are many ways to make code work (e.g., for loop vs. while loop), that there also existed practices to secure code (e.g., bounds check, type check). Similar to the early students, some intermediate students looked to ESIDE to help them make their code functional rather than secure their code.

Student behavior during the lab can be categorized into two behavioral areas. The first behavior exhibited by most students was a focus on project requirements. This group’s questions were targeted toward the short-term gain of “how” to make the code functional rather than the long-term understanding of “why” particular code does or does not work. The second behavior was exhibited by a smaller group who appeared to have a stable grasp of the assignment and tended to spend more time on interacting with ESIDE. This group also asked questions as to what ESIDE did, how it was accomplished, and why that was important.

ESIDE Interface

It was important to us to understand if ESIDE’s interface is intuitive and easy to use. Students reported that they found the interaction straightforward and simple to navigate. In general, we found that the warning icon did not “really disrupt coding” (participant no. 112) or get “in the way” (participant no. 86). However, a small group of students felt that the red color of the warning was a bit aggressive. After testing various colors (e.g., yellow, blue), we went with a darker red which was found to be more acceptable. Students also appreciated the ability to easily “go directly to a page that explained how to help” (participant no. 94). Overall, we believe ESIDE’s interface provides a simple yet robustly supportive interaction.

One drawback to our prototype is the use of Eclipse. Eclipse can be difficult to use and understand as it provides warnings that are too advanced for the students at these levels. Faculty have remediated this difficulty by conditioning their students to ignore available warnings. Despite this conditioning, students actively engaged with ESIDE’s warnings once encouraged to do so. To ensure future engagement, we modified the language used in the initial popups, so it could be read in a simplistic and inviting manner (e.g., “ESIDE believes you have a possible security issue with this code. Left click the red icon to find out why.”). Utilizing other IDEs may alleviate some of these issues.

Content Impression

A large challenge to complementing an existing course is the alignment of the supportive content. Materials that are too advanced or too early can have a negative effect. We therefore focused on the students’ interpretation of ESIDE’s content and interaction. All of the students reported that they understood that a problem existed when the warning icon appeared next to their code. As they interacted with the rollovers, popups, and educational pages, we discovered that about 80% felt that they understood the content as evident with the following statements concerning what they like about ESIDE:

“explained the issue in your code very well” (participant no. 214)

On the other side of the spectrum, there existed a group of students who had difficulty with the concepts related to the coding activity (e.g., how to create a while loop, what a file structure looks like). There was a tendency among this group to search ESIDE for classroom concept support, the effect of which ranged from a minimal time loss during the search process to further confusion. This confusion is most likely because ESIDE’s explanations assume that the student has a basic grasp of the classroom concepts. Their struggle became evident through observations and comments given as follows:

“I didn’t like that it was hard for me to figure out if my code was right” (participant no. 204)

Our remedy was to simplify and shorten the content found in ESIDE’s supportive layers. This included reducing the content in the initial popup box down to a short paragraph and altering informational pages so extensive explanations are only revealed when students explicitly expand sections. For example, we added an explanation regarding code for functionality versus code for security, which appeared to help students understand the difference in the empirical study described later.

Student Awareness and Use

Most students reported that they were unaware of the security issues related to the code they were writing. Overall, students reported that they liked the concept of ESIDE and supported the idea of having it available to them throughout their education. They felt as if the warnings were nonintrusive reminders of code issues and that they should investigate as to why they were receiving a warning. After working with ESIDE for just a short time, they became more aware of the larger security picture “it shows me that though my code will run with proper user inputs, it’s still vulnerable” (participant no. 134). Becoming aware of potential issues is the first step in helping students become aware of code-related security issues.

ESIDE has the capability of auto-generating secure code for known issues. This feature was available during the intermediate student study. While logged data were lost, students reported using this feature and appreciating its availability. This is quite promising as students were willing to secure their code. However, such a feature runs the risk of losing an educational moment as those who are task oriented might have the tendency to auto-generate code without reading the supportive content.

Results

ESIDE submitted a total of 45 interaction logs. As shown in Table 2, Section 1: Instructor A had 16 submissions, Section 2: Instructor A had 17 submissions, and Section 3: Instructor B had 12 submissions. From the logs collected, ESIDE provided an average of 29.2 warnings per student. The number of interaction logs, assignment code, and pre- or posttests vary as not every participant provided data for all aspects of the study (e.g., posttest was not taken).

OPEN IN VIEWER

Table 2.

ESIDE Interaction Results.

	Interaction logs (n)	Average warning per student	Interacted at least once	Interaction average	Accessed every instructional layer
Section 1: Instructor A	16	33.2	12 of 16 (75.0%)	2.63	5 of 16 (31.3%)
Section 2: Instructor A	17	22.4	10 of 17 (58.8%)	1.29	4 of 17 (23.5%)
Section 3 Instructor B	12	35.8	11 of 12 (91.7%)	3.00	5 of 12 (41.6%)
Total	45	29.8	33 of 45 (73.3%)	2.22	14 of 45 (31.1%)

Overall, 73.3% of the students interacted with ESIDE to some degree and 31.1% accessed every instructional layer. Interestingly, students in Section 3 had a higher interaction rate overall. This section also had their midterm exam on the due date of the assignment, and we postulate that students were using ESIDE as a review mechanism for their exam.

In looking at assignment code and student coding behavior, we found some unanticipated results. Both instructors reported that their students were struggling with arrays and that many had not turned in their assignments so much so that after our posttests and group discussions were complete, the instructors extended the deadline for any student who needed extra time. The instructors also reported that this was unanticipated as students in the previous semester did not exhibit similar behaviors on a very similar assignment. This would clearly have impacted ESIDE use as well and may have led to a reduction in implementing security techniques.

There was a clear pattern between sections when investigating whether students implemented a more secure solution in their code. As indicated by their professors, students had difficulty completing this assignment. Of the 43 collected, we found that only 34 had completed at least 90% of the work (Table 3). Of those 34, 13 had included secure coding practices, most of which were those who were in Section 1 who had secure coding requirements as part of their assignment. None of those with the midterm exam on the due date (Section 3) wrote secure code even though their level of interaction with ESIDE was greater than the other sections.

OPEN IN VIEWER

Table 3.

Code Behavior Results.

	Assignments collected	Complete assignments	Complete assignments with secure code written
Section 1: Instructor A	16	14	11
Section 2: Instructor A	15	11	02
Section 3: Instructor B	12	09	00
Total	43	34	13

In the open-ended survey questions, students reported similar results from the formative studies: an appreciation of the awareness of security that ESIDE provides and understanding of the content.

Knowledge: Pre- or Posttest Results

The knowledge tests were divided into two parts—questions on general software security as related to ESIDE and questions on specific secure coding solutions. The first section of students had a small amount of secure code requirements on their assignment. This section (n = 11) experienced an average pre- or posttest increase of 10.7 percentage points and with a secure coding awareness increase of 14.3 percentage points and a software security awareness increase of 4.5 percentage points. We conducted a paired-samples t-test (p < .05) and found a significant difference in the overall posttest scores (M = 73.55, SD = 6.07) and the pretest scores (M = 62.81, SD = 9.05); t(10) = 3.99, p = .002. Additionally, there was a significant difference in the secure code awareness posttest scores (M = 76.62, SD = 9.18) and the secure code awareness pretest scores (M = 62.33, SD = 13.99); t(10) = 3.31, p = .007.

Section 2 (n = 15) experienced an overall increase of 1.9 percentage points with a 2.0 percentage points secure coding awareness increase and a 1.8 percentage points software security awareness increase. No significant difference was found.

Section 3 (n = 9; the section with the midterm) experienced an overall increase of 11.1 percentage points with a 14.3 percentage points increase in secure coding awareness and a 5.6 percentage points increase in software security awareness. We conducted a paired-samples t-test (p < .05) to compare pre- and posttest scores and found a significant difference in the overall posttest scores (M = 78.78, SD = 4.29) and the pretest scores (M = 67.68, SD = 9.69); t(10) = 2.82, p = .022. Additionally, there was a significant difference in the secure code awareness posttest scores (M = 80.95, SD = 6.73) and the pretest scores (M = 66.67, SD = 9.52); t(10) = 3.00, p = .017.

Sections 1 and 3 experienced significant gains (p = .05) for overall increases and secure code awareness. Thus, sections with some incentives—assignment points or a midterm exam—seemed to have overall knowledge gains. However, our pre- and posttests are still in the early stages of development and due to the small numbers, results may vary as more data are collected. In addition, as can be noted on Table 4, all students scored fairly high on the tests, implying that correct answers were easy to guess.

OPEN IN VIEWER

Table 4.

Summative Study Pre- and Posttest Awareness Results, Reported as Percentage Correct.

	n	Pretest scores (code/concept)	Posttest scores (code/concept)	Percentage point gain (code/concept)
Section 1	11	62.81% (62.34%/63.64%)	73.55% (76.63%/68.19%)	10.74* (14.29*/4.55)
Section 2	15	69.48% (80.61%/50.0%)	71.43% (82.65%/51.79%)	1.95 (2.04/1.79)
Section 3	9	67.68% (66.67%/69.44%)	78.79% (80.95%/75.0%)	11.11* (14.29*/5.56)

Limitations

These studies were primarily qualitative explorations that encouraged students to interact with ESIDE and provide feedback in a group setting. Because of this environment, it is appropriate to be cognizant of possible limitations. The first limitation is the influence of the instructor and researchers in the room. While we believe the interaction was positive, this presence might have influenced the student interaction in either a positive (“I like xyz”) or a negative (reduced interaction) manner. A second limitation is the possible influence the group had on individual responses to group questions as many of the responses were positive and critical responses were minimal in comparison. A third limitation is the short exposure (less than 60 minutes) most of the students had with ESIDE, which means feedback is based only on initial impressions.

Another possible limitation is that Eclipse is a complex IDE that is not typical of entry-level Java courses (although it was used in all the courses we examined). Porting to other IDEs may provide even more usable support for students but would require substantial effort. Given these limitations, we believe our results provide support that ESIDE has the potential to influence secure coding practices throughout the semester and provides valuable lessons for improvements for longer and more general studies.

Timing

A critical component of ESIDE is the assumption that students have a basic understanding of the classroom concepts prior to complementing them with secure coding information. Because of this, it is imperative that ESIDE does not introduce new concepts too early as it might have a negative impact on the learning process and future ESIDE interaction. To this end, we designed our ESIDE studies to be aligned with expected student competency based on course progression.

There existed a small group of students who used lab time to learn classroom lessons rather than apply those lessons. Similarly, they explored ESIDE in search of class-related instructional materials as opposed to the available security-related materials. While the available materials did not necessarily confuse them, they did not find the functional support they were looking for.

Initially, we conceptualized ESIDE to be continuously available to students as they learn to code over the course of their education. However, these findings reveal the potential of abandonment as students might view ESIDE as being nonhelpful in many instances. In light of this new knowledge, we will need to explore how to control for timing. This could be done in three ways. The first is to have ESIDE ascertain the appropriate time to initiate interaction. The second is to provide the instructor with the ability to control when and what ESIDE provides the students. And the third is a means for students to assess and indicate when they are ready for support. These ideas will guide our future work.

Motivation: Incentives for Secure Coding

As can be expected, most participants revealed a task completion mindset that was primarily focused on the functionality of their code (Nance & Taylor, 2012). This is understandable as students are motivated to complete the requirements of their coding activities because successful completion is typically rewarded with higher assignment and exam grades. Previous researchers required students to attend a security clinic (for example, Belcher, Bishop, Dark, & Ngambeki, 2015; Bishop, 2010). However, we did not wish to require the use of our tool but instead examine whether minimal secure coding requirements on a weeklong assignment would influence the coding process and the students’ awareness. What we found are those students who had no incentive to learn about or write secure code (Section 2) interacted considerably less than their counterparts in the other two sections. Their interaction with ESIDE and their awareness scores were noticeably lower than the other two sections.

The evident motivator for Section 1 was the requirement to write secure code as part of their assignment. The higher level of interaction over Section 2 can be attributed to this incentive. This requirement also influenced coding behavior as 11 of the 14 completed assignments included secure coding. In addition, their secure coding awareness scores greatly increased while their software security awareness only had a minimal increase. This finding relates back to functionality. To expand, the initial layers of ESIDE support are related to the identification and remediation of secure code issues, while the last layer goes into depth as to why this is an issue which is where most of the software security awareness information is found. So, students spent time learning about the practices needed to make their code work as required and then they moved on.

Section 3 interacted with ESIDE at a level similar to Section 1 (points incentive) and also experienced a significant increase in their secure coding awareness scores. We have attributed this finding to an unforeseen variable which was the students using the assignment to study for the midterm exam which occurred the same day the assignment was due. When the students were provided with the assignment, the instructor informed them of its purpose as a review activity. To this end, we postulate that they spent more time exploring ESIDE for information concerning classroom concepts. Once they did not find said information, they moved on. This behavior is supported in their posttest scores as their secure coding awareness increased at a much greater rate than their software security awareness scores, which would have only dramatically increased if they had spent time exploring the final layer of ESIDE’s support.

These findings reveal the importance of the instructor’s role in guiding the student’s education. While we designed ESIDE to be as autonomous as possible, we have learned that we need to more deeply explore a variety of incentives for learning, both student and faculty driven, to encourage usage and maximize the potential usefulness of a tool such as ESIDE across a student’s educational experiences.

Future Work

These short-term studies provided positive evidence that our approach with ESIDE can work. Our results have led to continuous improvements in ESIDE, as well as directions for future design ideas. We need to explore methods for determining the proper timing and content of ESIDE for students in different courses. As a first step, we are currently adding means for faculty to control when their students are exposed to ESIDE content. We are also implementing methods for instructors to customize content, to allow them to more directly map classroom materials to ESIDE support. We are also exploring methods of alerting students that new content that they have not yet reviewed or used is available for them, which is currently not conveyed in the basic icons we have used in ESIDE.

We plan to continue to expand the coding constructs and vulnerabilities within ESIDE to broaden its use for more courses and support additional evaluations. There are many more questions regarding whether ESIDE can serve as an effective resource at multiple points and durations throughout a curriculum. For example, what will motivate students to continue to access ESIDE throughout the term? Will students become habituated to ESIDE and eventually abandon its support, within one course or as they are exposed to additional content in later courses? Will students incorporate ESIDE lessons into classroom practices over time? The ultimate goal is to ensure that computing students are graduating with sufficient knowledge and skills to prevent basic security vulnerabilities as they become professional developers, in turn, improving the security and robustness of the software they help to create.