Abstract
The role of technology is often propagated as an effective mechanism to expand access and opportunity in higher education, and this could not be more true than in the community college sector (Gierdowski, 2019; Tierney et al., 2018). A little over half of all students enrolled in U.S. postsecondary education are currently attending a community college (National Student Clearinghouse Research Center, 2017). This significant population represents an array of backgrounds and circumstances that drive community colleges to reconsider how they should deliver curriculum and student services (Bailey et al., 2015). Technology, in its many forms, has increased the capacity of community colleges to accommodate the multiple obligations (e.g., work, caring for a dependent) students balance as they pursue their educational aspirations. With the rising accessibility of smartphones (Apuke & Iyendo, 2018), students are able to search on-line for the information they need without having to step onto a college campus. Such a resource can be valuable to a student when their schedule or lack of basic necessities (e.g., funds for transportation) preclude them from visiting student services, using the tutorial center, or attending an instructor’s office hours. The ability to search for on-line content means that students have greater access to material to support their learning. Because on-line resources are often free, delivered in a wide array of formats, and satisfy a user’s immediate needs for information, they address common barriers to student achievement (Head & Eisenberg, 2010).
The vast amount of material available on-line has prompted researchers to understand how students sort and select, or evaluate, the results that emerge from their searches (Head & Eisenberg, 2010; Rieh & Hilligross, 2008; Wathen & Burkell, 2002). Since students depend on on-line material, as a resource, to facilitate their learning, understanding the basis of their process is imperative to how institutions develop more equitable resources and far-reaching strategies for student success. Given this context, the purpose of the current study is to answer the following question: When students are faced with several choices that emerge from their on-line search, what are the criteria used to evaluate and select resources that support learning course content?
This study draws on interview data from 12 students enrolled in a community college district, who offered insights on how they evaluated on-line resources for their science, technology, engineering, and mathematics (STEM) courses. We situated our inquiry within Michèle Lamont’s (2012) conceptualization of the sociology of valuation and evaluation to guide our protocol and analysis. Trust and utility were the prominent criteria by which on-line resources were evaluated. Students were skeptical of the accuracy of content in a given resource and used several dimensions of trust to direct their assessment. Students also evaluated with purpose, to search for and sort resources that reflected their goals and preferred conditions for engagement, or what we consider as utility.
The increasing role of the Internet in student learning reflects the growing discourse on meeting the diverse needs and circumstances of students (Hachey et al., 2013; Muse, 2003; Neuhauser, 2002; Pontes & Pontes, 2012). It is only appropriate then to focus attention on how students make sense of the growing availability of resources offered on-line. Identifying and understanding the criteria used to evaluate on-line resources expands scholarship on the relationship between students, technology, and access to opportunity. This is particularly consequential for the community college context in which advancing democratic tools, policies, and practices for diverse student populations remain crucial for how campuses live out their mission as open-access institutions (Gregory & Lampley, 2016; Johnson & Berge, 2012).
Literature Review
While the rise in postsecondary enrollment in the past century can be attributed to community colleges, completion and transfer rates, especially for minority and low-income students, are enduringly dismal (Schudde & Goldrick-Rab, 2015). Researchers have faulted the lack of structure in community colleges that differentially affect students’ experiences and outcomes (Bailey et al., 2015; Scott-Clayton, 2011). Community colleges widely function under a “cafeteria-style model” in which students select from a variety of resources and pathways (e.g., certificates, associate degrees, and transfer) without structured guidance because colleges assume that students possess the knowledge and experience to gather the relevant information and guidance to achieve their given goals (Bailey et al., 2015; Person et al., 2006). As a result, students, many of whom come from underserved communities, are overwhelmed by the vast number of choices and the difficulty of navigating pathway requirements and the complex and multi-layered structures of college campuses; this institutional condition is paired with insufficient coordinated information and support resources leading to negative consequences (e.g., such as selecting a course that does not satisfy a degree requirement) for students (Scott-Clayton, 2011).
The supply of information and other resources on-line to navigate college plays a significant role in how community colleges seek to mitigate the structural challenges that students often confront on their pathway to completion, although the benefits may be outweighed by the poor quality (e.g., ease of access, usefulness) of websites. The most obvious benefit being that students, depending on what they need and their possession of appropriate equipment (e.g., computers), can readily access this information without the burden of navigating in-person administrative procedures that, researchers have argued, are sources of inequality (Schudde & Goldrick-Rab, 2015). While on-line resources may be perceived as a solution, campus websites can extend the structural challenges associated with community colleges. A recent study, based on 20 Texas community colleges, found that campus personnel even expressed difficulty in locating and understanding transfer resource materials on-line, therefore suggesting that students are more likely to be confused about what they see on-line (Schudde et al., 2020). Community colleges working toward more structured pathways require that they understand the most effective ways to design and deliver information to their students. While research has established the importance of technology and on-line resources for students (Tierney et al., 2018), little is known about how students make sense of and evaluate this information.
Recent advancements in technology have altered the academic landscape for many college students. The use of technology in education has supported student learning (Dabbagh et al., 2019; Lai & Bower, 2020) and impacted learning outcomes (Baran, 2014; Calderón & Ruiz, 2015; Chan & Leung, 2014; den Haan & van der Voort, 2018). However, in the context of higher education, much of this research has focused on students at 4-year institutions (Kvavik, 2005). Studies that have looked at the community college environment reveal that students are using on-line resources to gain an understanding of a topic from class (Hanauer et al., 2004; Rennis et al., 2015) and to engage in on-line learning communities through social media (Fagioli et al., 2015). Such findings have treated on-line spaces as points of expanded access for content and social networking meant to facilitate student learning without regard to how students make the choice to engage. The current study places emphasis on a more fundamental issue that shifts attention to students’ process of evaluating the resources—to support their learning with course content—found on-line.
Given the breadth of information on the Internet, searching, collecting, and applying information for users can become a varied and complicated process (Hilligoss & Rieh, 2008; Rieh & Hilligoss, 2008). For community college students, many of whom may not be able to visit nor spend much time on campus, resources accessible through the Internet are essential to their academic success (Johnson & Berge, 2012). Access alone, however, does not mean that students will find the material to address their immediate needs. Students go through a process of evaluation in order to connect digitized material to their learning (Fogg & Tseng, 1999; Wathen & Burkell, 2002).
The malleable fabric of information that is susceptible to alteration and misrepresentation can lead students to be wary of their search results, and spend time evaluating the credibility of what they find (Rains & Karmikel, 2009; Rieh & Hilligoss, 2008). Information-seekers use five criteria to assess the credibility of on-line sources: accuracy, authority, objectify, relevance, and coverage, as seen in Figure 1 (Hilligoss & Rieh, 2008; Rieh & Hilligoss, 2008). Students consider several different components of a singular on-line resource are considered: esthetics, navigation, site ownership, and graphics (Lim & Simon, 2011; Rieh & Belkin, 1998, 2000; Slater & Rouner, 1996; Wathen & Burkell, 2002; Zhang & Duke, 2008). Those who are searching for information can develop a hierarchy of information of sorts where they will preference websites with .gov or .edu versus commercial websites (Zhang & Duke, 2008). This demonstrates that students are cognitively associating verified or government links as more “credible” than others, which potentially aligns with students’ perceived social capital and ability to preference and sort information (Beckman et al., 2018; Zhang & Duke, 2008).
Criteria to assess the credibility of on-line sources.
There remains the question of how students find value in the on-line resources and how they connect aspects of on-line resources to their learning. As evidenced by previous studies, the Internet is host to varied types of information that is communicated in an infinite number of ways. Students then must adapt to each different type of source (Zhang & Duke, 2008). This shifting focus for students calls for them to, first, have the ability to read and understand the type of information that is being communicated (Beckman et al., 2018; Zhang & Duke, 2008). Next, they must determine if said resource makes a cognitive connection with the various other on-line resources they may have also uncovered (Hilligoss & Rieh, 2008). Thus, the process of ascertaining the usability of an on-line source goes through an iterative process in which the information-seeker must be able to translate multiple sources and types of information (Beckman et al., 2018). The process becomes iterative when information-seekers stumble on contradictory information from a perceived trusted source. They will then cross-reference the content with people, materials from class, and other websites (Hilligoss & Rieh, 2008). This process, however, is not equal across disciplines (Calkins & Kelley, 2007). A student that has a background in one area of study may have a more challenging time seeking resources in other unfamiliar topics (Beckman et al., 2018). This may be due to their social and educational capital in being “trained” to read information a particular way (Zhang & Duke, 2008, p. 130). Distinguishing how students evaluate resources by disciplinary interest pushes against blanket findings on students’ engagement and Internet use.
Users require verification and reevaluation—a quick cognitive process for students—prior to using the on-line resource (Zhang & Duke, 2008). This process, however, has yet to be considered within the community college context, where students are faced with a lot of content on-line to sort and are more likely hindered by time constraints. The current study provides insight in the criteria community college students who aspire for a 4-year STEM degree, use to evaluate the material they find on-line for their learning. Research has yet to show how students “read with purpose” (Zhang & Duke, 2008, p. 130) and the conversation around how students are socialized to view the Internet as a source of “answers” remains untouched. The Internet contains boundless information. By focusing on how students evaluate on-line resources for their learning, the current study reveals how they make sense of the material. Asking students to identify the standards by which on-line resources are sorted and selected for academic use conveys the reasoning and purposes behind their evaluative process.
Conceptual Framework
When students turn to the Internet for material to improve their learning, they are faced with countless options that are not ranked, coordinated, or categorized in any meaningful way. Students, through the social and cultural process of evaluation, must make sense of this virtual heterarchy, a system that is inherently flat and unranked, and can have multiple “hierarchies of worth” (Lamont, 2012, p. 3). By tending to the phenomenon of evaluation, this study is focused on the criteria—of trust and value—that can drive how students organize on-line resources. We draw from Michèle Lamont’s (2012) conceptualization of the sociology of valuation and evaluation to guide our inquiry into what our participants consider (e.g., concepts, elements, and features) in their assessment of on-line resources.
Evaluation reflects and is constrained by the evaluator’s context, including their roles, knowledge, experiences, and beliefs (Lamont, 2012). Evaluation is a subjective practice in which an individual possesses their own references for which other entities are compared, negotiates criteria for evaluation against perceived sources of legitimacy that define dominant criteria, and constructs value by comparing and contrasting entities (Lamont, 2012). This process is undergirded by two sub-processes: categorization and legitimation. Categorization, in this instance, occurs when a student draws on previous learning experiences to compare and determine what the entity should be evaluated against, whereas legitimation draws on the student’s recognition of the entity as it relates to their knowledge of other’s perceptions of credibility. Moreover, these sub-processes can occur simultaneously. The process of evaluation then is both contextual and relational.
The design of the study takes Lamont’s theoretical considerations as our point of departure and guides us toward inquiring, drawing out, and analyzing the experiences and perceptions that shape how community college students evaluate and engage with on-line resources for their learning. While students must make sense and evaluate enormous amounts of information on-line, Lamont’s perspective suggests that students possess knowledge and experiences to determine the criteria used to evaluate the material that results from their search. In other words, Lamont’s conceptualization of evaluation meant that we anticipated students drawing on their knowledge, including sources of legitimacy and prior experiences, to undergird the criteria for evaluation. The findings in this paper elaborate on the knowledge and experiences that students bring to the evaluative process.
Method
The current inquiry stems from a larger qualitative study that examines community college students’ pathways to a 4-year STEM degree. Taking a constructivist stance, we collected interview data from 12 students across a single community college district on the West Coast. We designed the study to center our participants’ realities—in which they construct and assign meaning to the world around them—knowing fully that multiple truths exist and are needed to make sense of how students’ experiences and perceptions shape their process of evaluation (Charmaz, 2006).
The community college district hosts a National Science Foundation grant-funded program that supports the aspirations of low-income students in earning a 4-year degree in STEM. Students are awarded scholarship funds, assigned a faculty mentor, and given access to research opportunities and workshops during their time in the program. We invited all students in this program to participate in the larger study, which focused, primarily, on how their lives off-campus–such as family life, neighborhood, and work—influenced their capacity to earn a 4-year degree in STEM. At the time of data collection, the sample for this study represented 33% of all students in the program. Five students identified as female, one as non-binary, and the remaining identified as male. Seven students identified as a person of color. All students were either in their first or second year of taking courses needed for their application to transfer to a 4-year degree program in STEM. Data for the current study come from the larger study’s sub-focus on how technology plays a role in helping students navigate daily, structural challenges to educational achievement.
The primary mode of data collection was one-on-one interviews because it allowed us to tend to our participants’ “emotional dimensions of social experience” and “encourage [them] to evoke a variety of interactional settings, social contexts, and institutional situations” (Lamont & Swidler, 2014, p. 159–160). The latter point was especially important because, according to our theoretical framework, the criteria that participants used to sort on-line material were dependent on their context and experiences. The interview protocol included questions on their process for searching for on-line resources for their learning (see Appendix). Since the interviews were semi-structured, we asked additional questions that situated the participant in a particular context to distill when certain criteria were differentially prominent depending on the on-line resource. Interviews lasted 65 to 90 minutes and were conducted in a location of each participant’s choosing. Following Yin’s (2003) recommendation for ensuring validity, the design components of the study—including the questions, instrument, and analysis—were closely aligned and sensitive to our theoretical perspective on evaluation (Lamont, 2012).
A third-party transcribed the data. We took an inductive approach to our analysis and employed process coding to identify “action” and “process” (Saldaña, 2015). This resulted in two waves of line-by-line coding that included both pre-defined concepts, such as “legitimate,” and in vivo coding (Creswell, 2012). The primary investigator identified and recorded themes in the preliminary reading of the data. Two different researchers, then, coded the data separately, regrouped, and systematically delineated in vivo codes (Saldaña, 2015) in a table to be reviewed by the entire research team. After the second wave of coding, the research team deliberated discrepancies which lead to an examination for overlap and redundancy. The research team then collapsed codes into broader themes that demonstrated what on-line resources were evaluated for (LeCompte, 2000). The study is approved by the Institutional Review Board of the community college district, as well as by the home institution of the lead researcher.
Findings
On-line resources played a prevalent and democratic role in how participants succeeded in their studies. Multiple paths for learning and support (both in-person and on-line) are often available to participants in the form of office hours, tutorial centers, and support from fellow peers. Our participants’ engagement with them, however, was unequal, either because they were not effective in helping students learn or a student did not have sufficient time in their schedule. In contrast, on-line resources, such as those found through Google or YouTube, offered limitless possibilities to identify effective resources for supporting student learning, and represented few barriers to engagement as they can be accessed any time and, almost, anywhere. And yet, what makes on-line resources popular with students can also cause significant frustration. The resources students found on-line were described as “disorganized” and “too much to choose from.” The focus of this study was to identify the criteria participants used to evaluate on-line resources and the features that were perceived as useful in helping participants learn. As seen in Figure 2, we found that students sought to evaluate their trustworthiness—the extent to which the information was accurate and relevant—in a given resource by employing several criteria including university affiliations, public opinion, familiarity, and presentation. Participants also evaluated a resource for its perceived usefulness to their learning if it met their criteria for design and delivery. Although these criteria did not follow a temporal logic nor were they mutually exclusive, they were the dominant measures by which select on-line resources are allocated value, and therefore deemed worthy of the participants’ attention and focus.
Participant criteria and components for evaluating on-line sources.
Trust
Normally ranked the top visited sites on the Internet, Google, and YouTube were the most popular search engines for our participants. As students in STEM courses, participants often looked to websites or video clips as resources to solve a homework problem or better understand a concept that they had a hard time grasping in class or from reading the text. Depending on the key words used, searches yielded more than a million results. The issue of trust—evaluating the accuracy and relevance of the information shared on-line—was not taken lightly.
University affiliation
Symbols of legitimacy, such as a university logo, marked resources as more trustworthy as others. Participants looked to the website’s URL, credentials, and institutional affiliation. These markers often elicited feelings of certainty in the accuracy of the information shared. According to Tricia:
If it comes from a .edu website. . .I’ll click on those before the .coms because they are associated or they should be associated with an educational institution which have a pretty good track record of wanting to be helpful to students and accurate. It’s like medicine, it’s a profession you should trust with caution but generally trustworthy.
Websites that end with “.edu” are reserved for schools, colleges, and universities, institutions that are perceived to offer students the knowledge and skills to succeed. That assumption allows Tricia to narrow the available options and focus on the sites she has deemed valuable in light of the task at hand. The trust she places on “.edu” sites, however, has its limits, as alluded in her statement about the field of medicine. Like hospitals, it is generally understood that the quality of education can fall unevenly across schools and universities. Participants acknowledged a hierarchy based on credentialism and institutional reputation and rank to further determine which resources were more accurate, and thus more trustworthy.
Websites or YouTube clips hosted by professors or individuals with a PhD after their names were often given immediate consideration because occupation and level of degree attainment represented subject authority. Ariel shared that she has a list of favorites saved on her browser, “all three of the teachers that I watch [are] professors and [they] all [have] PhDs, they’re all actual professors.” The emphasis on “actual” demonstrates the distinction that Ariel makes between those with and without a PhD, suggesting that the notion of accuracy is closely related to the level of credential that the host of the video possesses. Lola goes further in distinguishing faculty and institutions by their reputation. She discussed the importance of using different criteria based on the subject matter:
This really good YouTuber he went to Berkeley. Berkeley is known for like a really good chemistry department. So of course I’m going to be like he’s legit, so I will take what he says.
For many participants, an “.edu” URL or a professor with a PhD was only the first step in their sorting process. Like Lola, participants considered the subject they needed support with and their knowledge of the institutions or individuals who would most likely offer accurate, or “legit,” content. Berkeley held the highest regard in the discipline of chemistry, which explained Lola’s tone of confidence and certainty in her response. The type of website, individual occupation and degree attainment, and institutional reputation were markers that helped participants direct and determine the level of trust in the accuracy of the information provided through the on-line resource. A resource was perceived as more accurate if it possessed multiple markers of legitimacy.
Public opinion
Public forms of expression, or recognition by others, influenced how participants determined the trustworthiness of on-line resources, especially if they were unfamiliar with the logic of institutional prestige and reputation. “Likes” and the number of times a website or a video has been visited or watched were initial critical measures in helping participants quickly sort through the many options that resulted from their searches. Erin, for instance, shared:
If you’re looking at lecture videos, you just look at the “Likes” in the video and how many people have watched it. . . some lecturers they have a lot of audiences, so that means that they are doing something right.
Participants associated these forms of expression to the accuracy of information shared through the resource. The higher the “Likes” or viewership, the more likely participants were going to place their trust in that website or video. These measures were especially useful in choosing resources among a handful of elite and reputable universities. If participants, such as Terrence, had the time, they would read through the comments that were located at the bottom of a site or that came after a video:
A lot of the comments, yeah they are either like positive or they are like yeah I get this, and then they either go through where they messed up at. So they kind of like go back to the steps that they did. They identify what they did wrong and they compare it to the video and yeah.
Comments offered participants additional context that cannot be derived from quantitative forms—“Likes” and viewership—of public expression. Participants were sensitive to “positive” comments, but often the comments that convinced them that the resource could be trusted were those from other users that discussed the usefulness of the information shared. These comments often laid out the participants’ experience in using the resource; for example, helping them identify and understand where in their process they went wrong in solving a problem. Participants can then determine if the resource in question offers accurate information. Trust, through the perception of accuracy, was derived from massive popularity.
Familiarity as reasonable
Participants compared the information shared through on-line resources against the information they gained in the classroom. Strong alignment between the two often meant the resource was deemed reasonable, which also alluded to its accuracy. Janae shared:
I’m like questioning all the time, whether or not like something is reasonable. Does this sound similar to what I learned in class? Like, does it sound like similar to the concepts I read from the book, like does it make sense of how they’re applying it, and do the numbers make sense too like. . .some of the small things are happening in between, some of these calculus concepts are super complex.
Participants looked to on-line resources because they wanted to support that could supplement the learning that took place in the classroom. For example, Janae discussed how certain mathematics faculty required students to solve problems using a particular method. In searching for additional resources, Janae had to make sure that the information shared through a resource sounded “reasonable” in light of her current learning. Were they employing similar concepts or methods? Any misalignment—as measured by the application of concepts or the reasonableness of solutions—may indicate that the information shared may not be accurate. Trust was constantly negotiated because the course structure (i.e., the content and delivery of content) may be their only point of reference, or the standard by which they can determine the accuracy of the information.
Contemporariness and clarity of presentation
How material is presented on-line makes a difference in how students perceive the material’s relevance. The criteria included the presence of contemporary visuals (e.g., images and use of colors) to signal newness and the clarity of the information expressed. Participants judged visuals and clear expression as a measure of relevance, which contributed to their decision to trust an on-line resource. Depending on the mode of the resource (e.g., website vs. a YouTube video), the criteria of presentation were used selectively.
University-hosted websites were popular among our participants. These websites held a variety of information and tools to strengthen a student’s mastery of a topic. If the site lacked contemporary and user-friendly visuals the relevance of the content was questioned. Janae elaborated on this point by sharing her experiences using on-line resources for her computer science class:
If it looks like Microsoft Vista 1990, I’m not going to take it seriously. . .I mean especially like in computer science, I took a Python class. Looking for like on-line computer science help, there are a lot of professors with websites. I’m like “Dude, you’re computer science professor, why is your website look like Java just threw up on it? I mean what is this?” It’s so old and then I can’t take it seriously either even though they probably are super smart but everything’s unorganized, the buttons are all weird. Keeping up with their websites is also keeping up with new ways to teach this technique.
Janae made it clear that websites with out-of-date designs can prevent her from trusting the relevance of the material (e.g., “technique”) presented. If the design of their website did not meet a participant’s perception of the relationship between relevance and contemporary design, the host professor’s reputation carried little weight. A website with a contemporary look meant that it must be relevant, or up-to-date. If the relevance of the material was questioned, participants perceived the resource as less reliable.
The clarity of content expressed also contributed to the participants’ evaluation of the on-line resource’s relevance to their course content. With the possibility of search results in the millions, not every item will be directly tied to what a participant needs at the moment. Barriers to that process of evaluation often meant that the trustworthiness of the resource would be questioned. The participant had to be able to read and/or hear the information clearly without strain. Lola recounts a website with a “red color background” and a “boxy font.” Both features made it difficult for her to evaluate the relevance of the information to her class. When it came to videos, Ariel shared:
Production quality is probably one of the biggest [criteria]. If I can’t hear them very well, their slides look like crap and there’s no good visual element on there, that will really steer me away.
A resource was not considered trustworthy, and therefore dismissed, if participants were unable to glean information from it. Even more so, according to Marie, “If the prose is funky and there are tons of grammatical errors or spelling errors I’m like okay. . .” Participants cannot determine what is relevant to them if they cannot even make sense of the material itself. Poor choices in design and expression were red flags for participants because they signaled challenges to their ability to evaluate the information provided.
Utility
Participants selected on-line resources for their capacity to strengthen their learning in a topic. Features of on-line resources that would indicate the utility of the resource were the design, specifically the navigability, of the on-line resource and the delivery of its content. This meant that participants were searching for on-line resources that were organized in a self-explanatory manner, and provided robust explanations, not simply solutions. Participants were focused on conceptually understanding topics for future application. And when it came to videos, instructor personality was key to how well participants received these explanations. In short, participants sought key aspects of design and delivery that they perceived as useful for their learning.
Design
The layout of the content within the on-line resource was a feature that demonstrated how well a participant could find the information they were seeking. Participants did not want to spend very much time searching and sifting. The material’s self-explanatory layout contributed to the perception of a resource as useful. Tricia appreciated YouTube videos with formats that were consistent and easy to navigate:
All of his videos have a very standard format so I know exactly what to expect and they are indexed really well. So I can find exactly the one I’m looking for with a simple search. And the videos are all him introducing what he’s going to explain and they are all just his computer screen. And then he’ll show an example or he’ll define it if necessary and then show an example and work it out.
For Tricia, consistency and navigability signal a sense of reliability and the on-line resource is evaluated favorably. Consistency was enshrined throughout, from the format of the video to how the instructor structured their delivery. Knowing what to expect meant that Tricia could spend less time searching for alternative resources. When promoted, participants could not describe what design aspects of these websites or videos would lead to easier navigation, but it was very clear for Tricia that “not putting the information in a format that works” for them was a reason to move on.
Along with signaling the contemporariness of the on-line resource, as indicated earlier, visuals also represented the quality of engagement that participants could expect. Marie explained:
The nicer something looks, like the nicer icon looks, the more I’ll be attracted to it. So if it’s a picture of a white board and handwriting, or if the photo quality is high or its very clear and organized for the same math problem.
Marie’s example reveals her understanding of the relationship between design and engagement. “Nicer” included using visuals—images or, even, handwriting—that signaled to easily digestible content. The right visuals can convince Marie that an on-line resource may be worth engaging. Visuals communicated to participants the likelihood that the resource would be useful to their learning because it was evidence that the material they were searching for reflected their preferred conditions for learning.
Explanations versus solutions
No participant utilized on-line resources to simply find solutions for their work. They sought out websites or videos because they wanted an explanation, a clear understanding of a process that led to the solution. This desire emerged from classroom experiences in which instructor lectures or assigned texts did little to advance participants’ understanding of the topic at hand. In recalling his experience in organic chemistry, Brian described why on-line resources were critical to his achievement:
I look on-line because they don’t explain “why” in the book, they don’t show you the process for all of it. I mean, they show you the concept and the initial process in the example stuff. But then sometimes that’s not enough for me.
The opportunity to search and sift through resources on-line was a reprieve from the constant struggle of finding support to learn course content that was difficult to grasp from lectures or course readings. But as Brian indicated, the utility of on-line resources is related to their provision of a path to finding and understanding the solution. As Alan further emphasized, “You don’t want just the answer, you want to understand what’s going on.” Websites or videos that were perceived as useful in this regard were described by Janae as “having very clear steps.” Having clear steps made it possible for participants to identify the stage(s) of solving a difficult problem, or validate that they were on the correct path. For Tricia, resources with clear steps are necessary, “So I can verify in my own mind that the path is correct.” Websites and videos that did not offer robust steps were often dismissed by participants because there was insufficient information to understand the topic or problem. Using resources that offered clear steps toward understanding a topic was perceived as more valuable for participants’ short- and long-term learning.
Instructor personality
Participants considered instructor personalities in videos across multiple platforms, such as YouTube, in their evaluation. The link between an instructor’s personality and participants’ engagement contributed to a video’s perceived usefulness. Personalities that were considered “engaging” often attracted participants’ attention. Ariel, for instance, expanded on this feature:
What’s funny too, the videos that are really popular among a lot of people is like that professor I was telling you earlier who’s really charismatic and love what he’s doing. The three people that I watch have similar trait. They’re warm, charismatic, they actually feel they love what they’re doing. . . Passionate too. And so I’m like wow this person is really excited about this molecule for some reason and I feel like I can be too.
Ariel highlights a common pattern that exists among the videos she views; they are published by individuals with similar personality traits that are attractive to her. The quality of one’s personality has an emotional effect, convincing Ariel that she, too, could be enthusiastic about the topic at hand. Instructors also reminded participants of prior positive experiences with classroom instructors, which was often the standard by which on-line instructors’ personalities were measured against. According to Bruce:
There is a professor from University of Arizona. For each of his classes, he does like a version of it in about 16 minutes long. And it’s so cool. He’s funny, he’s really on top of it and he just speaks just like my professor. Like he uses the same notation and all that stuff, almost the same problems too.
The instructor in Bruce’s example displayed personality traits that spoke to his capacity to engage and learn. Participants had different preferences in defining video instructors they deemed engaging. Bruce mentioned, “funny,” whereas other participants preferred those who expressed “confidence” in their presentations. Prior context, however, often played a role in how participants negotiated their decision to use the on-line resource. Seeing similarities in personality between his prior instructor and the video instructor offered Bruce comfort in using the on-line resource that he described.
The features of trust and utility were prominent in our participant’s process of evaluating on-line resources for their academic learning. Participants turned to on-line resources since they are easily accessible. The cognitively laborious process of sorting and sifting through on-line resources, however, is time-consuming, and it can be frustrating for participants because of the vast amount of disorganized, and inherently unranked, information that resulted from their searches. Our findings show that students evaluated on-line resources for their capacity to be trustworthy and useful for their learning. Participants identified various features of trust and utility that were unique to some, and common across different mediums such as personal websites and YouTube videos. These features reflected the contextual and relational nature of evaluation in which students drew on prior cultural knowledge and learning experiences, and were influenced by the perception of others via “Likes” and public comments.
Discussion
Our 12 participants offered insight into how they sorted and selected on-line resources for their learning in STEM classes. Unlike previous studies that have framed on-line spaces as mechanisms for expanded access to content and social networking (Wathen & Burkell, 2002), our findings place a necessary emphasis on how students choose to enter those spaces in the first place. Lamont’s (2012) conceptualization of the sociology of valuation and evaluation, through the sub-processes of categorization and legitimation, shaped the current study’s capacity to identify and understand the criteria that underlined our participants’ process of evaluating on-line resources. We asked the following question: When students are faced with several choices that emerge from their on-line search, what are the criteria used to evaluate and select resources that support learning of course content?
We found that the vast amount of resources on-line leads participants to spend time evaluating their potential to be trustworthy and useful for participants’ academic learning, as seen in Figure 2.
Our findings imply that students evaluate on-line resources with intentionality. Although the Internet serves as a crucial gateway for participants to access resources both on- and off-campus (Shea & Bidjerano, 2014; Venegas, 2007), accessibility does not necessarily lead to engagement. Trust and utility for academic learning took many forms, in that these forms reflect the contextual and relational experiences and perceptions that developed the parameters for evaluation. The current study both confirms and builds on prior research as seen in Figure 3.
Criteria and components for evaluating on-line sources.
First, our findings demonstrate and reinforce that accuracy and legitimacy were key criteria in trusting a resource, and perceived forms of authority (e.g., a URL with a “.edu”) helped users create a hierarchy in an inherently unranked space (Beckman et al., 2018; Rains & Karmikel, 2009; Zhang & Duke, 2008). University affiliation was a prominent feature that revealed participants’ belief of colleges and universities as legitimate stewards of knowledge (Olaisen, 1990; Slater & Rouner, 1996). This is similar to Schudde et al.’s (2020) finding on community college staff encouraging students wishing to transfer to visit university websites for more reliable information if they are unsure of where to start. However, the extent to which legitimacy is a clear criterion for students is contingent on their background and familiarity with, for example, reputable colleges and universities. While the study did not inquire about participant backgrounds, future work should clarify their relationship with the criteria used for evaluation, thereby providing more nuanced insights that can improve usability. Previous experiences were also points of reference for users to gauge any contradictory information they encountered in their process, which prior research has confirmed (Hilligoss & Rieh, 2008; Lamont, 2012). The current study makes two distinct contributions to understanding how resources are evaluated for their trustworthiness.
First, an indication of popularity through “Likes” or positive comments can bias how users perceive the content, which begs the question, “Is something popular because it is actually good, or is it popular just because it is popular?” (Chang, 2013; Muchnik et al., 2013). Our participants expressed greater attraction to a resource because of public displays of opinions. This prompted participants to feel confident, a result of the influence of “herd mentality” (Chang, 2013), in their choices to engage. With many sites offering users mechanisms to publicly evaluate the content, future studies should consider the effects of a herd mentality on student comprehension and achievement. The contemporary presentation of the resource and the clarity of its content were also associated with notions of trust, which was built on the belief that a more up-to-date design, free of typos and misspellings, would mean that the material was relevant and accurate. This is an important finding as visuals are often associated with “design,” which is often treated as a criterion for access and utility (Rieh & Belkin, 1998, 2000; Rieh & Hilligoss, 2008). We show that trust can be located across many on-line features. As products and content evolve, and access to the Internet becomes more prominent at community colleges, future inquiries should revisit and monitor how students find trust during their on-line engagement.
Second, selected resources were structured and delivered in a useful manner that aligned with the participant’s purpose and preferred conditions for learning. Perceived utility was a key criterion by which resources were also evaluated. Participants looked for ease of navigation, including the ability to sort through and identify specific content within a vast website of information, or to make sense of how topics are organized. Participants may be less likely to engage with an on-line resource if the desired information is difficult to locate, or if the website does not appear up-to-date (Schudde et al., 2020). We confirm that esthetics, navigation, and graphics are prominent criteria for which on-line material is evaluated for engagement (Rieh & Belkin, 1998, 2000; Rieh & Hilligoss, 2008). These findings add to the literature by illustrating that utility is also defined by a participant’s intended goal and preferences in instructor personality.
The current study conveys how participants “read with purpose” (Zhang & Duke, 2008, p. 130). Context and user goals are critical in how the evaluation of on-line resources is understood. Our participants provided insights that revealed how their role as community college students guided their process for evaluation. By situating Lamont’s conceptualization of evaluation as two sub-processes of categorization and legitimation within the community college context, we were able to identify new criteria that add to the scholarship of how students engage and make sense of on-line resources.
Students can struggle to locate, organize, and make sense of information they need to curate informed choices for their learning and academic plans because many community colleges struggle to provide students, amidst the multiple options and degree pathways, with clear and structured paths (Rosenbaum et al., 2006; Scott-Clayton, 2011). Community colleges are also resource-constrained, which creates challenges for staff to invest time and reduces the institutional capacity to ensure that students are able to effectively locate reliable information (Schudde et al., 2020). While improvements at the organizational level (e.g., updating websites) may take time, students’ needs for immediate and accessible information will continue to grow. The current study’s findings suggest that when making these improvements, campuses should consider both how students make sense of on-line resources and training students on how to search effectively for accurate and relevant information, a skill that will likely be useful beyond their time on campus.
Limitations
Limitations from this study should serve as a point of departure for future research. First, our inquiry is but a snapshot of our participants’ relationship with on-line resources. As they gain new skills in searching for on-line material, or encounter new forms of on-line resources, their evaluative process may change. Future research should consider monitoring this process over time to glean patterns that can more clearly reveal the relationship between student learning and achievement and the Internet. Second, our study allowed participants to describe any on-line material that they evaluate and engage with. The criteria that we identified may be more prominently located in some forms of on-line resources over others. Future research should develop inquiries that either focus on a single type (e.g., YouTube videos) or draw insight from comparative work (e.g., YouTube vs. Twitch). Third, evaluating is an iterative process that depends on a user’s specific situation and broader context (Dervin, 1992). Findings from this study are specific to the participants in STEM fields in community college. Future research should look into what measures students use to evaluate on-line resources in different academic fields as a means to build on-line spaces that are more compatible with how students read with purpose.
Implications
The community college landscape is changing rapidly to accommodate the diverse needs and goals of students. On-line/hybrid classes, for instance, are a growing phenomenon meant to increase access and opportunity. In light of these changes, colleges and universities have yet to make meaning of how students turn to the Internet for additional support and learning. As higher education seeks to become more responsive to student preferences, understanding how students go about selecting and engaging with the Internet is crucial. Based on these implications, below are three recommendations for community colleges, and potentially all higher education institutions, regarding access to technology and literacy for on-line resources that can shape and facilitate students’ learning and ultimately, academic success.
Campus-Wide Recommendations
Since many community college students spend time much of their time off-campus because of work and/or responsibilities at home, they often engage in remote learning. Although students are accessing information through tablets and smartphones, a reliable and sturdy piece of equipment critical to students’ learning is a desktop or laptop, which has more capabilities than mobile devices. Institutions cannot assume that everyone has access to desktops or laptops at home. Policies should be developed to ensure students can borrow laptops for a period of time outside of school and have wi-fi access. While some colleges have created these policies—usually housed in an institution’s library facility—returns are confined to the library’s business hours. This may not work for community college students, whose schedules often preclude them from visiting campus during that time. By investing in equipment like laptops and hotspots, and expanding borrowing policies that reflect the diverse realities of their students, institutions can address unequal access to technology and equipment.
Student Affairs Recommendations
This study has revealed that evaluating the credibility of an on-line resource is a complex multi-step process that is neither cognitively linear nor formulaic. Instead of community college students having to figure out which on-line resource to use and why, institutions can delineate general steps for this process that have empirically been assessed. Communication about this information to students should be accessible, engaging, and quick to read; for example, a student resource center can create an infographic with main themes, short descriptors, and images with examples to save students time and cognitive labor in assessing the credibility of an on-line resource. This study’s findings demonstrate how students were thinking about which resources to use (e.g., YouTube videos, message boards, or forum threads) with specific criteria in mind, and how they tested the accuracy of the on-line resource, which varied by discipline. Resources that outline steps in evaluating on-line resources can also be tailored to specific majors like those in STEM fields. This information should be made available electronically and in paper format for students who visit student resource centers.
Classroom Pedagogy Recommendations
On a classroom level, instructors can support students’ use of technology and on-line resources by integrating the practice in their pedagogy. For example, when introducing a new concept, instructors can use a video to supplement the lecture. The introductory lessons in STEM are foundational for grasping more complex information in later courses, and this significant time is pivotal for students who may need various ways of conceptualizing the new content. By integrating an instructor’s use of on-line resources, instructors can also model and explain why they have chosen to use this particular on-line resource. Instructors should include links to helpful websites, articles, and videos in the syllabus, in addition to links found in textbooks, so students can save time searching and sorting through the vast number of on-line resources.
Our participants were either in their first or second year of college, taking many of the gateway courses to transfer into a 4-year degree program. These courses are key to helping students build a firm foundation for future learning and achievement in science (Gasman & Nguyen, 2019). Failing these courses may lead to dire consequences, including declined interest in STEM or attrition from both STEM degree pathways and college (Wang, 2015). On-line resources play an immense role in helping students succeed. Our work suggests that staff and instructors build in programmatic or classroom time to help students manage the process of searching and sorting these resources. Although this did not occur for our participants, they did speak of peers that used questionable resources (e.g., Yahoo! Answers) that often lead to greater confusion. Improving student success in STEM should take into account how students make evaluate on-line resources.
Conclusion
The Internet serves as a digital reservoir for infinite amounts of information. For many, turning to this endless bank of data begins a personal process that brings to light a wealth of experiences and clarity for the individual learner. For our participants, using on-line resources to further their learning and academic achievement represents a core component of their educational experience in higher education. As the pedagogical foci continue to evolve in community colleges, on-line and asynchronous resources—and how students make meaning of these items—is an immeasurably crucial point to understand when expanding and changing future curriculums. This study extends the focus on credibility to begin delineating the aspects of on-line resources that students find most beneficial. Understanding how students sort and evaluate on-line resources offers insights into a learning environment increasingly defined by the Internet and informs how institutions and instructors might better incorporate these resources into their curriculum and academic supports. The emergence of on-line resources and modified learning environments are not new in higher education. However, the many channels of information-seeking and sharing must first be better understood in order to build conditions that facilitate the academic success of every learner. Adult learners can easily become lost within the limitless information available on the Internet, but it is the duty of a collective higher education support to assist students in making sense of and understanding all they have found.
Footnotes
Appendix
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is funded by the National Science Foundation (Award #: 1643580).
