A Survey and Analysis of College Students' Understanding of Planet Formation Before Instruction

3.1. Setting and participants

This survey was conducted at the University of Arizona, a public university located in Tucson, Arizona. In 2016, undergraduate enrollment exceeded 34,000 students. Approximately 52% of the undergraduate population is identified as female and 48% is male. Slightly more than half (51%) of the undergraduate students are Caucasian, 26% are Hispanic, 5% are Asian, and other ethnicities make up 17% of the population. Less than 1% of students reported that their ethnicity was unknown. At this university, 71% of students are in the age range of 18–22 years (The University of Arizona Fact Book, 2016–2017).

All the participants in this study were undergraduate students enrolled in introductory astronomy or planetary science courses. Students enrolled in these courses are typically nonscience majors taking the courses to fulfill their General Education requirements (Prather et al., 2009). At the University of Arizona, undergraduate students are required to take two 100-level (Tier 1) science courses and one sophomore-level (Tier 2) science course. Introductory astronomy and planetary science courses are popular and so often fulfill these students' Natural Sciences requirement. Astrobiology is available as a Tier 2 option. We surveyed students in both Tier 1 and Tier 2 courses, but due to the introductory nature of the material taught at both levels, we categorize them both as “ASTRO 101” for the remainder of this work.

Due to the required nature of these courses, students enrolled in ASTRO 101 are typically in the first 3 years of their undergraduate tenure. The demographics of the students enrolled in these courses are consistent with the University of Arizona's undergraduate population as a whole. To conduct educational research with human subjects, the University of Arizona requires approval from the institutional review board (IRB). This study has been approved and classified as “exempt,” meaning the project does not pose any harm to the students participating in the study and is not subject to further review. ^*

ASTRO 101 courses at the University of Arizona typically enroll anywhere from 50 to 150 students (the Tier 1 courses generally have higher enrollments). These courses aim to provide students with an appreciation for the size, scale, and structure of the Universe, in addition to providing instruction on a variety of basic topics such as Moon phases, the Solar System, the nature of light, and stellar evolution (Slater et al., 2001). Typically, these courses are taught using a traditional lecture-based format.

3.2. Instrument development

To determine the topics for the SSR questions, we conducted a preliminary analysis of 27 syllabi and lecture slides (when available) from undergraduate introductory astrobiology courses taught predominantly in the United States. At first, we surveyed only astrobiology courses because planet formation is commonly taught in these courses as a precursor to exoplanetary systems. To expand the data set, we then requested syllabi and lecture slides from instructors with any experience teaching planet formation in an introductory course regardless of the course title. After this request, we analyzed the content of seven additional courses, leading to a total of 34 courses surveyed. An analysis of the syllabi, lecture slides, and written notes showed that the following subtopics were most commonly addressed when teaching planet formation:

The nebular theory (gravity and angular momentum).

Based on these findings, we developed six different SSR questions that incorporated elements from these topics. The final list of SSR questions is presented in Table 1.

SSR Question 1 was the most general of the short-answer questions and was developed so we could explore common themes and misconceptions that appeared when asking students to explain the overall process of planet formation. SSR Questions 2 and 6 probed students' ability to describe and explain the architecture of planetary systems. We developed these questions because it is important for students to understand their solar neighborhood and the layout of our planetary system before they can comprehend the compositional and structural differences between our Solar System and those discovered around other stars.

SSR Questions 3 and 4 covered the topics of planetary motion and the definition of a planet, respectively. We expected that college students would have previous exposure to these topics from high school or middle school, and thus, we wanted to evaluate how well they understood these more basic concepts. According to the physical science content standards from the NSES (NRC, 1996), position and motion of objects is recommended for study as early as the K-4 level. The concept of motion is to be emphasized with the inclusion of forces at the fifth to eighth grade levels and reiterated again at the high school level (grades 9–12).

College-aged students were typically between the ages of 7 and 10 years when the International Astronomical Union (IAU) modified the definition of a planet to the one that is currently upheld: a planet must be in orbit around the Sun, it must be massive enough for its self-gravity to lend to a [nearly] spherical shape, and a planet must clear its orbit of any debris (Meeting of the International Astronomical Union, 2006). The third and final criteria led to the demotion of Pluto to dwarf planet status. Although students are aware that Pluto is no longer a planet, the intent of SSR Question 4 was to probe whether or not students have an understanding of the current definition of a planet, and whether or not they are able to differentiate planets from other celestial objects (e.g., stars, comets, asteroids, moons).

SSR Question 5a was developed because when developing the TOAST, Slater (2014) reviewed the content of three expert position statements that discussed the most critical topics in astronomy. The NSES, developed by the NRC (1996); Project 2061: Benchmarks for Science Literacy, developed by the American Association for the Advancement of Science (1986); and the American Astronomical Society's Goals for ASTRO 101 (Partridge and Greenstein, 2004), all converged on the idea that the evolution and structure of the Solar System was one of the most important topics to discuss in introductory astronomy courses. SSR Question 5a evaluated students' understanding of the basic definition of a solar system.

After coding SSR Question 2, it was clear that many students were unable to correctly explain the structure of our Solar System and many were unable to differentiate between the Solar System and the Universe more generally. As a result, Question 5a asked students to provide a definition of a solar system in addition to what objects they would expect to find there.

SSR Question 5b asked students to provide the definition of an exoplanet. In the past 22 years, we have discovered more than 3700 planets that orbit stars other than our Sun (NASA Exoplanet Archive, 2018). These exoplanetary systems are often at different evolutionary stages than our own Solar System, and can thus shed light on the planet formation process. We can use exoplanetary systems with Jupiter-sized planets orbiting on very short orbital periods, for example, as evidence for planetary migration during solar system formation (Armitage, 2010). Articles in the popular media commonly discuss the discovery of new exoplanets, particularly those with characteristics similar to Earth orbiting in the habitable zone around their host star(s). Due to the relatively recent introduction of exoplanets into the ASTRO 101 curriculum, as well as their extensive coverage in the popular media, we developed SSR Question 5b to explore students' basic understanding of an exoplanet.

3.3. Data acquisition

Students in 13 ASTRO 101 sections were asked to respond to one of six SSR open-ended survey questions relating to the topic of planet formation. Students answered only one question to ensure they would give a quality response while not taking up too much instructional time. This allowed us to survey the greatest number of students from eight Tier 1 and five Tier 2 courses to determine if the response quality was greater from individuals who had taken a previous astronomy or planetary science course. Since the goal of this work was to determine college students' understanding of planet formation before instruction, we administered the surveys during the first week of the Fall 2016 and Spring 2017 semesters before any relevant material was taught.

In the Fall 2016 semester, we administered SSR Questions 1–3, and in the Spring 2017 semester, we administered SSR Questions 4–6. Although the SSR questions had multiple parts, students were typically able to complete their answers in less than 10 min. The SSR questions were randomly distributed among the courses and among the students within each course. Furthermore, the surveys did not request any information that would allow individual students to be identified. Each student answered one short-answer question and provided information regarding whether or not they had taken any previous astronomy courses. By the end of the two semesters, we had received responses from a total of 1050 students. The number of responses to individual questions ranged from 167 to 192.

Students were not required to participate in the survey. Due to the anonymous nature of the survey, however, we do not know exactly how many students declined to participate. A comparison of course enrollments to the number of responses we received implies that fewer than 5% of students in attendance when the surveys were administered declined to participate.

3.4. Data analysis

Once the surveys were collected, they were analyzed using a post hoc coding process consistent with grounded theory (Glaser and Strauss, 1967). Our coding procedure closely followed the procedure outlined in Bailey et al. (2009). First, one of us (the lead author) read through every response to each individual SSR question and recorded common themes, ideas, and misconceptions. These were vetted with two additional education researchers as an independent check of the data. These themes, ideas, and misconceptions were then assigned a specific code, and this process continued until no additional themes emerged from the data set. The frequency of each code was recorded. Many of the student responses were lengthy, and one response could often be coded for multiple themes. Since each question typically tackled a different aspect of planet formation, the six questions were coded separately.

Many of the broader themes that appeared in the data set required subcategories (Bailey et al., 2009). For example, when analyzing responses to SSR Question 1, we found that many students mentioned that the Solar System formed during the Big Bang. We created a larger umbrella code entitled “Big Bang” and then created a subcategory code “After Big Bang” to note how many students were able to correctly identify that our Solar System's formation was not coincident with the formation of the Universe (Table 3).

When analyzing each of the six SSR questions, we noted how many students had taken a prior astronomy course, how many responses we were unable to code due to the quality of response, and how many students answered the questions with phrases such as “No Idea” or “I Don't Know.” These responses were assigned the codes “Prev Astro,” “Not Codable,” and “No Idea,” respectively.

We were unable to code responses if the student did not attempt to answer the question in any meaningful way. For example, when one student was asked to describe the characteristics of the planets in our Solar System, and whether or not planetary composition changes with location (SSR Question 2), the response was “There are many unique planets in our solar system. Each planet has something of their own that another planet does not. Each planet is effected in their own unique way when dealing with location and distance from the sun” (SSR Question 2—Student #182). Responses we were unable to code, and those given the code “No Idea,” were not included when determining the frequency of each theme. Thus, \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}{ \rm{Codable \ Responses}} \,=\, { \rm{Total \ Responses}} \\\; - \left( {{ \rm{Not \ Codable}} \, + \,{ \rm{No \ Idea}}} \right)\end{align*} \end{document}

For more details, see the headers of Table 2 versus Table 3. Reponses we were unable to code differed slightly from those coded as “miscellaneous.” Responses were determined to be miscellaneous when the student attempted to meaningfully answer the question, but their response was not consistent with the larger themes identified from the data set (Tables 9 and 14).

After coding the responses to each question, we performed a second level of analysis. At this stage, students' responses were compared to what would be considered a full credit, correct response on a final examination after learning about planet formation in an ASTRO 101 course. To generate these “correct” responses, we enlisted the help of an associate professor of Planetary Science at the University of Arizona who researches planet formation and teaches this topic in ASTRO 101 courses. The professor's responses served as the upper anchor of student responses and were written in such a way as to mimic a student-level response.

Once the full-credit responses were provided by a professor who is a content expert, each student response was put into one of four possible categories as outlined in the study of Bailey et al. (2009): Correct (C), where the response was complete and did not contain any incorrect statements; Incomplete (I), where the response was missing one or more of the components necessary for a full-credit response; Partial (P), where the response was partially correct but also contained incorrect statements; and Wrong (W), where no part of the students' response matched any component of the full-credit response.

For SSR Questions 1, 2, and 4, an additional category was introduced: True but Insufficient (T). This category was used when a response included true statements, which were off-topic, so they did not answer the question in any significant way (Bailey et al., 2009). Responses that we deemed not codable were either classified as “Wrong” or “True but Insufficient” depending on the content of the response. Students who responded “No Idea” were classified as “Wrong” for this component of the analysis.

At first, it was not intended that any of the SSR questions would cover the basic definitions of planets, solar systems, and exoplanets. After coding the first three SSR questions from the Fall 2016 semester, however, it was clear that many students were using these fundamental terms incorrectly, and questions probing their understanding of these definitions were required. As a result, we developed SSR Questions 4 and 5, and administered just SSR Questions 4, 5, and 6 in the Spring 2017 semester.

4.1. SSR Question 1: general knowledge of planet formation

A total of 170 students responded to this question in the Fall 2016 semester, and 150 responses were classified as codable (Table 2). Approximately 44% of those responses attributed solar system formation to the Big Bang in some capacity. Around 15% of students who mentioned the Big Bang claimed that the Solar System formed after the Big Bang, but only two students were able to provide the correct timescale, billions of years after. A substantial fraction (36%) of students mentioned the process of material coming together to form planets, which was coded as “accretion” despite the fact that no student actually used the word accretion in their response. Nearly the same percentage (35%) of students stated that gravity or a “pulling force” played a significant role in the formation of the Solar System.

Almost one quarter (∼22%) of students coupled the themes of accretion and gravity. A small percentage (∼5%) of students provided more complex responses that were coded for accretion, gravity, and collisions. One of these students stated, “A bunch of rocks drawn into the orbit of the Sun collided to make the rockier planets. The more gaseous planets were probably formed when a bunch of gas, possibly from the remnants of a dead star, got together because of gravity. I really don't know what specific raw materials were the origin, or where they came from exactly. Basically, a bunch of space stuff got together because of gravity” (SSR Question 1—Student #17).

Another student affirmed that, “Lots of cosmic dust starts colliding after a super nova/Big Bang creates a gravitational pull. More and more collide and the celestial objects grow. This continues for billions of years, eventually the clumps get big enough to attract everything near by, sort of ‘cleaning it up’ and eventually these [clumps] are big enough to be considered planets” (SSR Question 1—Student #140). Although these responses were not entirely correct, since these students were able to identify that collisions lend to accretion, and that the force of gravity plays a major role in the accumulation of material, these higher-level responses had components consistent with an accurate description of planet formation.

Of the 150 codable responses, 11 students (7%) discussed more advanced topics when asked to explain how planets form. For example, one student responded, “It started as a flat disk from a nebula. The rocky planets formed closer to the middle because the material could withstand hot [temperatures], while gaseous planets formed past the freeze zone” (SSR Question 1—Student #136). These student responses were not complete, but they did introduce higher-level concepts that were found in only a small percentage of responses. Interestingly, of the 11 students who mentioned more advanced topics in their response, only 1 student response was also coded for gravity, accretion, and collisions. This finding highlighted that although students may utilize more advanced terminology, they still lack a total understanding of the planet formation process. A more complete list of themes can be found in Table 3.

For SSR Question 1, a Correct response included gravity, collisions, and accretion in addition to the basic understanding that gas giants and terrestrial planets follow slightly different formation processes in terms of what material is predominantly accreted. Only three students were able to incorporate all the components necessary for a Correct response. A typical response to SSR Question 1 was “The Solar System was created by an explosion called the Big Bang—resulting in the formation of our planets in the Solar System” (SSR Question 1—Student #67), and since nearly 50% of students mentioned the Big Bang in regard to planet formation, these responses were predominantly classified as Wrong. The classification breakdown for the entire question is presented in Table 4.

4.2. SSR Question 2: planetary composition and the architecture of our Solar System

SSR Question 2 asked students to describe the composition and characteristics of the planets in our Solar System, along with whether or not planetary composition changes with location. A total of 192 students responded to this question in the Fall 2016 semester, and 187 responses were classified as codable (Table 5). Approximately one quarter (27%) of these responses stated that the rocky planets were closer to the Sun, while the gaseous planets were further away. A smaller fraction (15%) of students stated that there was a region with abundant ice even further from the central star.

This question had multiple components, and many students chose to answer only part of the question and left the other components blank. However, 94% of students answered the component of the question that asked them to explain the composition of the planets in our Solar System. The majority (68%) of students mentioned that the planets are made of a combination of rocks, asteroid fragments, and dust particles (all categorized as solids). Additionally, 60% of students stated that planets are made of gas, and 37% of students more vaguely stated that planets are made of small particles (e.g., molecules, elements, and atoms).

Sixteen students (9%) answered that planets are a combination of rock and gas, and an additional 16 students went a step further and claimed that planets are made of a combination of rock, gas, and ice. Nearly one quarter (24%) of students separated the planets into two categories: rocky planets and gas giants, while 5% of students separated the planets into three categories: rocky planets, gas giants, and ice giants.

When asked whether or not planetary composition changes with location, 79% of students replied “Yes.” Less than half (47%) of students provided an explanation as to why planetary composition changes with location, but those who did most commonly attributed planetary composition change to the amount of heat received from the Sun at each planet's location (25%). Only three students (2%) correctly attributed the transition from rocky to gas giant planets to the presence of the snow line.

The most complete student response to SSR Question 2 was “The inner planets Mercury, Venus, Earth, and Mars are the terrestrial or rocky planets. They are composed of rock. The outer gas giant planets Jupiter, Saturn, Uranus, and Neptune are composed of ice and gas—mainly gas with icy cores. Composition changes with location because in the inner solar system, during formation, it was too warm for ices, so the planets formed from rock and accreted small gaseous atmospheres. In the outer solar system, past the snow line, the presence of ice allowed the planets to form bigger icy cores and accrete much larger gaseous atmospheres” (SSR Question 2—Student #190). This student response was significantly more complex than a typical student response to SSR Question 2, which was “Planets are made of rock and gases. Yes, [their composition does change with distance from the Sun] because the amount of heat they receive changes with distance” (SSR Question 2—Student #151). A complete list of the common themes identified in each component of SSR Question 2 can be found in Tables 6 –9.

For this question, a Correct response required students to distinguish rocky planets from gas giant planets and ice giant planets. Students were also required to note that rocky planets were located closer to the Sun than the giant planets. Finally, a complete answer included an explanation of the snow line and its role in separating the rocky planets from the gas giants. Since this question was particularly complex, only 2% of students answered the question completely. A majority of the responses were classified as Partial (51%) since almost every student was able to provide some explanation about the basic composition of the planets in our Solar System. The classification of all the student responses to SSR Question 2 can be found in Table 10.

4.3. SSR Question 3: planetary motion and migration

A total of 168 students responded to this question in the Fall 2016 semester, and 166 responses were classified as codable (Table 11). More than half (57%) of these responses correctly identified that the planets in our Solar System orbit the Sun in the same direction, and an additional 10% of students went on to say that these objects orbit in the same direction but at different speeds. Only one student was able to connect planets orbiting in the same direction to the planet formation process. This student stated, “Yes [planets orbit around the Sun in the same direction]. Due to the planetary formation process, in which planets form out of a swirling vortex of dust, gas, and ice around a new star, they orbit in the direction this vortex was spinning” (SSR Question 3—Student #168).

Six students (4%) mentioned that the planets orbit around the Sun on elliptical orbits, a concept that is taught at both the middle school and high school levels. A large majority (84%) of students affirmed that the planets likely did not form in their present locations, and nearly three quarters (73%) attempted to provide an explanation. The most common explanation (28%) was that planets were pulled into their current orbits due to the force of gravity, or the misconception that planets could not have formed in their current location because objects in space are constantly moving as the Universe is expanding (22%). A small percentage (8%) of students alluded to the concept of planetary migration, while an even smaller percentage of students attributed planetary motion to collisions with large objects (7%). A complete list of the common themes identified in each component of SSR Question 3 can be found in Tables 12 –14.

A Correct response to SSR Question 3 included a statement that planets orbit the Sun in the same direction on elliptical orbits. Additionally, Correct responses involved an understanding that planets likely did not form in the exact locations they are in now and that planetary migration was responsible but not well understood.

There were very few Correct responses to this question because even though nearly 10% of students alluded to migration in their responses, only 4% of students mentioned the elliptical nature of planetary orbits. It was surprising that twice as many students mentioned migration as mentioned the elliptical orbits. This could have been due to the phrasing of the question, and if this question was to be administered again, it would be beneficial to ask students about the shape of planetary orbits more explicitly. A majority (61%) of responses to this question were classified as Partial. The classification of all the student responses to SSR Question 3 can be found in Table 15.

4.4. SSR Question 4: basic understanding of a planet

A total of 167 students responded to this question in the Spring 2017 semester, and 155 responses were classified as codable (Table 16). More than half (57%) of these responses mentioned that a planet must orbit a sun or star. Nearly one quarter (23%) of students defined a planet in terms of its physical composition, claiming a planet must be made of rock or gas. Nearly the same percentage (22%) of responses stated that a planet must be a certain size, and 10% of students stated that a planet must be bigger than an asteroid, comet, or moon. Although a substantial percentage (19%) of students mentioned that a planet must have a distinct orbital path, only six students (4%) stated that a planet must clear its orbit.

One example of a particularly comprehensive answer to SSR Question 4 was “A planet is an object that is formed during the creation of a solar system that is large enough to become spherical by its own gravity and has cleared its orbit of other objects. It is different than other objects like the Sun because it isn't large enough to start fusion. Asteroids and comets aren't large enough to become spherical by its own gravity and they haven't cleared their orbits of other objects. Planetoids such as Pluto are large enough to become spherical but they haven't cleared their orbits of other objects” (SSR Question 4—Student #146). Although this student had not yet taken a previous astronomy course, they were able to correctly pinpoint the reason Pluto is no longer classified as a planet. Furthermore, this student was able to provide scientifically accurate explanations when distinguishing a planet from other objects in the Solar System, in addition to correctly mentioning keywords and phrases such as “spherical” and “self-gravity.”

Another student directly addressed Pluto's demotion to dwarf planet status by stating, “A planet is a satellite around a star that is large enough to clean up the area around its orbit. For instance, Earth has cleaned up the surrounding materials while Pluto has not. This is why Pluto is not considered a planet” (SSR Question 4—Student #151). These responses were significantly more detailed than a typical response to SSR Question 4, which was “A planet is a mass that orbits around a star” (SSR Question 4—Student #64). Of the 155 codable responses, only 3 (2%) of them addressed every component of the working IAU definition of a planet. A complete list of the common themes identified in SSR Question 4 can be found in Table 17.

A complete response to SSR Question 4 included that a planet revolves around a sun (star), that it is massive enough to be roughly spherical in shape, and that a planet clears its orbit. Furthermore, a complete answer also discussed at least one characteristic of a planet that separates it from other celestial objects (e.g., planets do not fuse hydrogen, planets revolve around stars while smaller bodies revolve around planets, smaller bodies are not necessarily spherical in shape).

The majority of responses were classified as either Partial (33%) or Incomplete (22%), but a large percentage (40%) of responses were classified as Wrong. Responses were typically marked Wrong if students neglected to mention any of the components in the working IAU definition of a planet and instead mentioned that planets must, for example, have an atmosphere, have moons, or have an environment potentially sustainable for life. The classification of all the student responses to SSR Question 4 can be found in Table 18.

4.5. SSR Question 5

SSR Question 5 covered two distinct topics: the definition of a solar system (SSR Question 5a) and the definition of an exoplanet (SSR Question 5b). The original intention was to analyze both components of SSR Question 5 together, but only 60% of the students who were given SSR Question 5 attempted to answer 5b. Due to the distinct nature of the topics covered and the variation in the number of students who answered each component, we coded Questions 5a and 5b separately as shown in Tables 19 –26.

4.6. SSR Question 6: Solar System formation and its impact on planetary system architectures

The goal in developing SSR Question 6 was to take the themes addressed in previous SSR Questions and apply students' knowledge of planet formation to solar system architectures. A total of 178 students responded to this question in the Spring 2017 semester, and 172 responses were classified as codable (Table 27). This question was the most difficult to code since the range of answers varied significantly, and there was no “typical” response to this question. Approximately one quarter (26%) of students stated that gravity “helped” our Solar System form. A non-negligible percentage (19%) of students went on to explain in more detail that closest to the Sun, the gravity was strong enough to pull the denser rockier planets in. These students held the misconception that because terrestrial planets have higher densities, they are more massive than gas giant planets, and a greater deal of gravity was required to pull them into their current orbits around the Sun.

Nearly one fifth (17%) of students stated that the Sun's temperature directly affects the layout of our Solar System, but could not state a particular reason. Ten students (6%) asserted that the layout of the planets in our Solar System was directly related to the order in which they formed. Only four students (2%) mentioned the snow line in their responses. One of these students asserted, “[The layout of our Solar System] tells us that planets inside the frost line formed from colliding rocks and metals and the gas giants formed outside the frost line collecting ices and hydrogen compounds that stayed frozen due to the distance from the Sun” (SSR Question 6—Student #113). When asked if all solar systems have to mirror the layout of our own, 34% of students said “Yes,” 55% said “No,” 9% of students did not provide a response, and 2% of students said “Both Yes and No.” A complete list of the common themes identified in SSR Question 6 can be found in Tables 28 and 29.

Similar to SSR Question 2, a Correct response to SSR Question 6 included that the distinction between rocky and gas giant planets in our Solar System is due to the location of the snow line and the condensation temperatures of different elements during planet formation. Students were also required to mention that due to the differing condensation temperatures of metals, rocky minerals, and hydrogen compounds, we would expect that (in other solar systems) gas giant planets would be located further away from the central star while rocky planets would be located closer in, like in our Solar System. Due to the complexity of this question, if students said that the layout of other solar systems would likely follow a similar trend to our own, their responses were classified as Correct even if they did not provide a fully correct explanation.

One student who had not taken a previous astronomy course provided the response, “The idea right now is that the rocky planets formed close to the Sun because that was all that could survive the hotter temperatures. The gas planets formed further away where it was cooler. For the most part, gas planets have to form further out, but we now know that a bit of them move in closer to their star (e.g., Hot Jupiters)” (SSR Question 6—Student #91).

Science majors in upper level astronomy courses would be expected to know about exoplanetary systems with Jupiter-sized planets on orbits well within the orbit of Mercury, and be able to attribute this architectural difference to planetary migration in gaseous disks. Students in 100- and 200-level courses were not required to make such interpretations to have their responses marked Correct. For this question, only two students provided Correct responses. The vast majority (85%) of responses were classified as Wrong. The classification of all the student responses to SSR Question 6 can be found in Table 30.

5.1. Summary of significant results

We used the SSR questions listed in Table 1 to compile and analyze the frequency of (n = 1050) students' ideas on the topic of planet formation during the Fall 2016 and Spring 2017 semesters, before relevant instruction. The most common misconception students' had was that nearly 50% of students who answered SSR Question 1 asserted that our Solar System either formed directly from the Big Bang or as a direct result of the Big Bang. These findings were consistent with previous studies that analyzed fundamental misconceptions in astronomy (Prather et al., 2002; Simonelli and Pilachowski, 2004; Bailey et al., 2012; Wallace et al., 2012).

Furthermore, when probing students' understanding of the Big Bang specifically, Prather et al. (2002) found that 80% of college students asserted that the Big Bang was an explosion of pre-existing matter. We found that nearly 20% of students who answered SSR Question 1 attributed the formation of the Solar System to a large-scale explosion (although we did not ask students to explain whether or not they believed matter existed before the Big Bang), which is a physically incorrect characterization of the Big Bang (Wallace et al., 2012).

College students' commonly held belief that our Solar System formed from the Big Bang demonstrates that ASTRO 101 instructors need to provide students with a lesson on cosmological time before teaching planet formation. Students must understand that the formation of the Universe and the formation of the Solar System are independent events taking place on vastly different scales and separated in time by 9 billion years.

Over one third (36%) of students who were asked to describe the general process of planet formation (SSR Question 1) were able to correctly identify that planets form by accretion. Although students did not explicitly use the term “accretion,” they were able to describe a process of material coming together due to the force gravity to form planets (see Section 4.1). Students described accretion as the accumulation of small particles (25%), the accretion of rocks and asteroid-sized bodies (10%), the accretion of gaseous material (8%), or some combination of the three. Nearly one quarter (22%) of students asserted that accretion occurs due to the force of gravity (a pulling force) driving chunks of material toward each other.

Nearly 10% of students who answered SSR Question 1 and 8% of students who answered SSR Question 6 claimed that the Sun's gravity specifically acts as a catalyst for accretion. Since the Sun makes up the vast majority of our Solar System's mass, students tend to view the presence of the Sun as a significant factor in determining a planet's gravity (Williamson and Willoughby, 2012). When explaining the process of accretion, instructors should make it clear that a planet is massive enough to have its own gravity, and it is the gravity of the growing planet that attracts surrounding material to be accreted.

SSR Question 6 also illustrated that many students have a difficult time distinguishing mass and density. Nearly 20% of students who answered Question 6 claimed that the rocky planets in our Solar System are closer to the Sun because they are denser, more massive, and thus experience a larger force of gravity. One student stated, “After the Big Bang, more dense and larger/more massive planets did not travel as far as those which are ‘lighter.’ This is explained with the gravity formula where larger mass means more attraction, holding ‘heavier’ planets closer to the Sun and ‘lighter’ ones further away” (SSR Question 6—Student #153). Another student claimed that “Since the rocky planets are more dense and weigh more than the gas planets, they gravitate towards the Sun” (SSR Question 6—Student #108). These responses clearly demonstrate that students assume that higher density means higher mass without taking a planet's size into account (Williamson and Willoughby, 2012).

In the case of our Solar System's planets, the gas giant planets are significantly more massive than the terrestrial planets despite the fact that gas is less dense than rock, and in fact they almost certainly have more massive rocky cores than the terrestrial planets. This can be attributed to the fact that in the outer Solar System past the snow line (where the gas giant planets form), the surface density of solids increases by a factor of ∼3, and planets are able to grow large enough to accrete a gaseous envelope (Kennedy and Kenyon, 2008). The combination of large icy and rocky cores with significant gaseous envelopes makes the gas giant planets more massive than the inner rocky planets.

Students' inability to distinguish mass and density can lead to an incorrect understanding of the role of gravity. This misconception is particularly difficult to address since the density of an object indirectly provides information about an object's mass (Williamson and Willoughby, 2012). As a result, students are not only conflating density with mass, but also misapplying the gravitational force law. In ASTRO 101 courses, it is important to clearly define mass and density before delving into the gravity equation. Furthermore, it is important for instructors to state that gravity is directly proportional to a planet's mass, and that the inner planets do not have more gravity just because they are located close the Sun. To do this, instructors could simply use our Solar System's planets to illustrate how higher density does not necessarily equal larger mass, and by extension, a stronger gravitational force.

SSR Question 2 explored students' understanding of planetary compositions and solar system architectures. An in-depth understanding of planetary composition is required before students are able to fully comprehend the process of planet formation and its application to planetary systems beyond our own. Most students were able to correctly identify that planets are made of primarily solid rocky material (68% of responses), gas (60%), and ice (26%). It is particularly interesting to note that more than one quarter of students who answered SSR Question 2 mentioned ice as a primary component of a planet's composition, as “the notion of grouping planets as rocky, gaseous, and icy is relatively new to the field of astronomy” (Plummer et al., 2015, p. 1390).

When asked to explain if planetary composition changes with location, and why, less than half of students provided a response, and those who did primarily attributed compositional differences to the amount of heat given off by the Sun at each planet's location. Students who answered SSR Question 2 (and those who answered SSR Question 6) affirmed that the Sun's temperature affects the layout of our Solar System. What they fail to understand is the fact that it is the condensation temperature of elements rather than the Sun's temperature that governs our Solar System's architecture. Although students are correct that closer to the Sun the temperature is hotter, they seem not to understand that planetary composition is primarily determined by the condensation of refractory elements (like rocks and metals) at high temperatures and volatiles (e.g., hydrogen compounds) at lower temperatures past the snow line (Lodders, 2003).

Although condensation temperature is not a trivial concept to teach, it is important that the condensation of elements and the role of the snow line are discussed when teaching planet formation at the ASTRO 101 level. Otherwise, students will continue to leave ASTRO 101 courses with a superficial understanding of the physical characteristics of the planets and be unable to explain the layout of our Solar System. This will limit them from being able to draw any parallels between the composition and locations of the planets in our own Solar System and those that are being discovered around other stars.

Additionally, SSR Question 2 emphasized students' inability to take basic concepts introduced in ASTRO 101 and apply these concepts to explain more detailed processes or phenomena. SSR Question 2 was broken down into four parts for analysis (Tables 6 –9). When students were asked, “What are the planets [in our Solar System] made of,” 94% of students provided a response. When asked to explain why or why not planetary composition changes with distance from the Sun, however, less than half (47%) of students attempted a response.

Furthermore, with the exception of SSR Question 5b (on the topic of exoplanets), the questions that had the largest percentage of responses classified as “Wrong” were SSR Questions 1 and 6 (Fig. 1). These were the two questions that required students to apply concepts like gravity, planetary composition, and temperature to explain the more complex process of planet formation. These findings shed light on the fact that there needs to be an emphasis on the application of physical principles in ASTRO 101 courses and not simply an overview of basic astronomy concepts.

OPEN IN VIEWER

FIG. 1.

Classification breakdown for all six SSR questions. Question 5 was split into two parts: 5a and 5b for coding purposes (see Section 4.5). Questions 1 and 6, which tested students' understanding of complex processes, and Question 5b, which tested students' understanding of exoplanets, had the largest percentage of responses classified as Wrong. Question 5a, which tested students' understanding of the basic definition of a solar system, had the largest percentage of responses classified as Correct and Incomplete when compared to the other questions. Questions 2, 3, and 4 had a substantial percentage of responses classified as Partial. SSR, student-supplied response.

As previously mentioned in Sections 4.4 and 4.5.1, SSR Questions 4 and 5 were administered in the Spring 2017 semester after coding SSR Questions 1–3. It was clear from the Fall 2016 responses that many students were unable to differentiate a planet from a star, and others believed that the galaxy (or nebulae) exist(s) inside our Solar System. When asked to specifically define a planet, 3% of students who answered SSR Question 4 claimed that a planet and a star were analogous, and only 5% clearly stated that a planet must be smaller than a star. Bailey et al. (2009) found that 6% of students asked to respond to the question, “Is there a difference between a star and a planet?” selected “No,” and an additional 10 students (∼5%) selected “both yes and no” for their response.

ASTRO 101 students have a difficult time comprehending the basic definitions of celestial objects, as well as their size and scale relative to each other. This is made even clearer when analyzing students' understanding of the relationship between the Solar System, Milky Way Galaxy, and Universe (Simonelli and Pilachowski, 2004). Bailey et al. (2012) found that among nearly 200 introductory astronomy students, 52 (26%) of them provided an incorrect response when asked to describe this relationship, and students often confused solar system and galaxy. Of the 172 codable responses to SSR Question 5, 13 students responded that they expected to find galaxies or nebulae within our Solar System, and 7 students deliberately stated that the Solar System is a region that includes the Milky Way Galaxy.

Although these percentages (8% and 4%, respectively) were not as high as those found by Bailey et al. (2012), there were additional student responses that mentioned the Solar System consists of many stars (34%), including specifically the constellations that make up our night sky (2%). It is crucial that instructors not assume that students are able to distinguish celestial bodies from each other or that they have an even basic understanding of these definitions. Before lecturing on the topic of planet formation, instructors should provide an overview of basic definitions, as well as spend time discussing the size and scale of the Universe. Nonscience majors do not have an intuitive grasp of the huge range of scales of time and space encountered in astronomy, so they need initial orientation.

SSR Question 5b asked students to define an exoplanet at the most basic level. As mentioned in Section 4.5.2, only 60% of students who answered SSR Question 5 even attempted to provide an answer to the exoplanet component. This was particularly surprising, given the high visibility of exoplanet discovery over the last decade. Furthermore, the discovery of potentially habitable exoplanets is often publicized on social media and online news platforms. Students' inability to even attempt SSR Question 5b highlights the need for professors to teach topics that are at the forefront of current astronomical research (Pasachoff, 2002). Since these topics appear in magazines and news reports, it is in the best interest of ASTRO 101 professors to help their students understand the significance behind these discoveries.

5.2. The impact of previous exposure to astronomy

Of the 1050 students surveyed in the Fall 2016 and Spring 2017 semesters, 221 had taken a previous astronomy course at the high school, college, or community-college level. One limitation of this work was that we did not ask students if they had studied planet formation in their previous astronomy course, so we were unable to track students' retention of the material. Since nearly one quarter of students who answered SSR Questions 1–6 had taken a previous astronomy course, we compared their level of understanding with students who had never seen the material at higher than an elementary school level. Nearly one tenth (9%) of students who had taken a previous astronomy course provided a response to their respective SSR question that was classified as Correct. Students who had not taken a previous astronomy course provided a Correct response 4% of the time.

Approximately one quarter (22%) of students who had previous astronomy background knowledge provided Incomplete responses. A smaller percentage (15%) of students who had not taken a previous astronomy course provided responses that were classified as Incomplete. Interestingly, nearly identical percentages of students provided Partial credit responses (36% and 34%) whether they had previous astronomy or no previous astronomy, respectively. Students with no previous astronomy courses contributed a significantly larger percentage of responses classified as Wrong (44% vs. 31%). Although the classification “True but Insufficient” was not commonly used for either group, students who had not taken a previous astronomy course were twice as likely to provide a True but Insufficient response. A graphical representation of these percentage differences can be found in Figure 2.

OPEN IN VIEWER

FIG. 2.

The percentage of students from each category (previous astronomy vs. no previous astronomy) whose responses were classified as Correct, Partial, Incomplete, Wrong, or True but Insufficient. Students who had taken a previous astronomy course were more than twice as likely to respond correctly to their given SSR question when compared to students with no previous astronomy background. Students with no previous astronomy course (at higher than an elementary school level) typically provided responses that were classified as either Wrong or Partial. Due to the complex nature of SSR Questions 1, 5b, and 6, even with previous astronomy knowledge, student responses were still typically classified as either Wrong or Partial, although there was a greater percentage of Incomplete and Correct responses amongst the six SSR questions from students with previous astronomy coursework.

It is not surprising that taking a prior astronomy course yielded a greater percentage of Correct and Incomplete responses when compared to students who were not familiar with the material. A portion of the SSR questions did explore students' understanding of topics taught at the elementary school level (the definition of a planet and a solar system, planetary orbits, gravity, etc.). Despite previous exposure to some of the material, the small fraction of Correct and Incomplete responses from both groups emphasizes instructors' need to incorporate classroom practices that promote retention of material from an early age. Furthermore, the limited number of Correct and Incomplete responses highlights the complex nature of the topic of planet formation, and the need to better familiarize students with the physical concepts behind the creation of their solar neighborhood.