Using YouTube Playlists for Qualitative Research: A Classroom Activity for Content Analyzing Video Ads

Introducing Content Analysis to Undergraduates

The traditional mix of research methods very often introduced to marketing and advertising students includes survey design, one-on-one interviewing, and focus groups. These are the standard “big three” most frequently or often covered in academic courses (Freeman & Spanjaard, 2012). Less common but nonetheless important for practitioners are other more qualitative-leaning research methods, such as ethnography, case studies, grounded theory, projective techniques, and content analysis (Freeman & Spanjaard, 2012; Thyroff, 2019). While it is true that content analysis is often characterized as a quantitative method or a mixed-method technique, the activities of analysis and interpretation so central to many advertising research tasks, such as understanding trends or monitoring competition, provides warrant for characterizing content analysis as a more subjective and qualitative research method (Spiggle, 1994). In the last 20 years (roughly), disciplines in the social sciences as well as information and library science have explicitly differentiated qualitative and quantitative approaches to content analysis (Berg, 2001; Zhang & Wildemuth, 2017). Activities, for example, that involve reconciling or explaining observed phenomena or those research tasks which require generating new ideas, theories, or explanations about a considered data set are more aptly described as qualitative than quantitative. Quantitative methods, generally described, involve an approach that is more often aligned with hypothesis testing, defeating prior conclusions and rendering conclusions from literal or manifest content.

In the marketing and advertising disciplines, definitions for content analysis tend not to invoke the qualitative-quantitative distinction. Following Kassarjian (1977) and Kolbe and Burnett (1991), there is a focus on “objectivity” and “systemization,” as methodological requirements, and core notions of validity, reliability, and coder-consistency remain salient. Three helpful and now well-known definitions of content analysis that stem from the field of consumer behavior are as follows:

• “Content analysis is an observational research method that is used to systematically evaluate the symbolic content of all forms of recorded communications” (Kolbe & Burnett, 1991, p. 243).

Common to all three definitions is mention of a technique or method that aims at objectivity in the rendering or interpretation of gathered data or communications. “Objectivity,” in this context, is contrasted with “subjectivity,” “noise,” and “bias:” elements of a research instrument or measure that obfuscate or undermine the purpose or goal of the analysis. Insofar as a content analysis, as a process, adheres to strictly defined rules and protocol, the research method can offer some guarantee for validity (i.e., a norm or standard for researchers to affirm that they are measuring what they think they are measuring) (Fico et al., 2008).

Systematization and a process involving a series of judgments with trained judges or coders are also common elements of the three (above) definitions. Rules and constraints for systematizing the process for analysis allow for reliability in how the data or content is processed. This is sometimes called or referred to as test–retest reliability (Riffe et al., 2019). Test-retest reliability is a form of methodological consistency, a necessary condition for validity, and provides stability in time, stability in place, and stability for each independent observer (Fico et al., 2008, p. 123).

The psychological notions of “latent” and “manifest” content, used in the third definition (above), are also important and prove useful for analyzing advertising and communications media (Fico et al., 2008). Whereas manifest content is described as “denotative,” generally regarded as possessing a shared meaning (e.g., considered easy to recognize, easy to count or quantify), latent content requires more “reading between the lines,” and both rules and description are needed to consistently recognize or pick out variables or items of interest (Fico et al., 2008, p. 120). Allan (2008) provides a good example from the academic advertising research literature of both manifest and latent variables in his study of 3456 prime-time television commercials. See Table 1 for a brief historical overview of how content analysis has developed as a research method in the marketing and advertising disciplines, beginning with Lasswell (1951) and concluding with the role of AI and machine learning for the research method (Lee et al., 2020).

OPEN IN VIEWER

Table 1.

Introducing Content Analysis to Undergraduate Advertising Students: Reference Studies.

Description	Title/Source	Author/Year
Often cited as one of the first researchers to utilize content analysis, Lasswell was a political scientist who analyzed text in European newspapers to track and understand the rise of totalitarianism during the 1930s.	The Analysis of Political Behavior: An Empirical Approach Routledge publishing	(Lasswell, 1951)
A classic in the marketing literature, this article reviews definitions of content analysis and provides guidance on “objectivity,” “systematization,” and “quantification” as methodological requirements.	“Content analysis in consumer research” Journal of consumer research	(Kassarjian, 1977)
Reviews mechanisms for establishing reliability and objectivity in content analysis research. Builds on Kassarjian (1977).	“Content-analysis research: An examination of applications with directives for improving research reliability and objectivity” Journal of consumer research	(Kolbe & Burnett, 1991)
Summarizes why consistency is important for content analysis and reviews several content typologies that are utilized in advertising research.	“A SAS MACRO for calculating intercoder agreement in content analysis” Journal of advertising	(Kang et al., 1993)
Provides succinct criteria for analyzing audio-based variables. Includes good examples of how video ads can be categorized.	“A content analysis of music placement in prime-time television advertising” Journal of advertising research	(Allan, 2008)
Provides an eight-step process for conducting a qualitative content analysis. Two example research projects (external to advertising) are described.	Social Research Methods to Questions in Information and Library Science Libraries unlimited	(Zhang & Wildemuth, 2017)
A thorough resource providing book length step-by-step instructions on the use and application of content analysis. Useful for extending content analysis to other research contexts.	Analyzing Media Messages Routledge university press	(Riffe et al., 2019)
Reviews how AI technology is being used in content analysis. The appendix to this article includes good examples of codebook formatting for a variety of variables and units of analysis.	“Making sense of text: Artificial intelligence-enabled content analysis” European journal of marketing	(Lee et al., 2020)

YouTube in the Classroom

Murphy and Bouchacourt (2020) reports that short and focused videos can be an effective teaching tool in the advertising classroom. Five-minute YouTube videos that profile topics and luminaries in the field of advertising in the Stan Talks YouTube video channel are shown to be effective tools for classroom discussion and student research projects (Murphy & Bouchacourt, 2020). The stability of the platform, particularly its accessibility from a mobile device or smartphone, makes it an ideal learning tool in a variety of classroom settings due to accessibility and existing familiarity.

Cowley (2020) describes a learning activity that uses YouTube to explore digital marketing skills in video creation and search engine optimization. Two lessons, transpiring over roughly two weeks, allow students to engage with YouTube’s search algorithm and experience how keyword and competitive research can lead to improved organic search results (Cowley, 2020). YouTube and third-party keyword tools are explored in an active learning environment where outcomes are contingent upon real-world end-user interactions.

Pace (2008) explores how YouTube can be used in a consumer research context and as a way to catalog and evaluate consumer narratives. Storytelling, videography and narrative discourse are elements of an interpretive framework where insight is derived from consumer-generated videos (Pace, 2008). Patterson (2018) also explores videos of self-expression posted by YouTube users to gain sociological data on biracial relationships (Patterson, 2018). The usefulness of YouTube’s auto-transcript feature, where audible language is transcribed into text, proves particularly useful when analyzing manifest variables (Patterson, 2018). Recognizing YouTube as a video-sharing repository, social media channel (where end-users comment and interact) and a platform application with various tools, such as video editing, transcription, adding clickable links, and creating video playlists, presents many opportunities for advertising students and educators alike.

Pedagogical Strategy

The basic strategy of the classroom exercise we describe incorporates students’ existing procedural knowledge of YouTube with a seven-step process where students can practice working together on a content analysis of video ads Figure 1. As Ambrose et al. (2010) concludes, “Students learn more readily when they can connect what they are learning to what they already know” (p. 18). Placing students in teams of three to four individuals prompts active learning and course engagement. Very often, even if one particular student lacks prior knowledge of YouTube or one of its features, there will be another student with sufficient know-how to fill in any gaps. In addition, if there are any teams in the class that are essentially starting from scratch with creating YouTube playlists, YouTube has an extensive how-to video library that can quickly catch teams up to speed.OPEN IN VIEWER

We recommend teams comprised of three individuals: one person who is designated as the “Playlist Compiler” and two others who take on the job or responsibility of “Coder.” In odd-numbered classes, four-person teams can be formed simply by assigning two people to the role of Playlist Compiler. Although all team members will need to form a consensus and agree on the research goals and parameters of the content analysis, dividing up the tasks in terms of playlist compiler(s) and coders, respectively, helps guarantee roughly equal time spent out-of-class preparing and participating in the learning activity. Three class periods are recommended to complete the introduction, review the seven steps (outlined below), and report each team’s findings. Whereas smaller class sizes are more friendly to concluding oral presentations from each team, larger class sections are perhaps more amenable to written reports whereby the instructor simply aggregates and summarizes findings on the third/final day of the lesson. In addition to the logistics of dividing the class up into teams, the introduction stage of the learning activity can also include an overview and definitions of content analysis as well as brief examples of how researchers often move back and forth between concept development and data collection during the early stages of a study (Zhang & Wildemuth, 2017). Variables or units of analysis found to be important for audiovisual studies in advertising, it can be emphasized, are often discovered and refined in the early stages of the data gathering process. Explaining how this type of flexibility and openness are typical and useful for content analysis research helps level-set expectations with novices who are just beginning to work with the research method.

The automobile industry is a convenient industry to practice content analyzing video ads. Ads are readily available on YouTube for multiple product lines for multiple brands, and although not necessarily timeless, they are relatively stable for the foreseeable future. Furthermore, the automobile industry is fairly mature, garnering consistent ad spending year over year from various international brands and remains one of the more consistent verticals that invests in video and TV advertising. Sample research questions, variables, codebook definitions, and a classroom worksheet are provided below with automobile industry ads in mind. Other possible industries well suited to meet our classroom activity, however, could include general liability insurance (home and auto) as well as pharmaceuticals. Both industries have a diversity of brands, and video ads are quite prolific on YouTube. Home and auto insurance commercials, in particular, are ripe for classroom analysis when identifying various types of humorous appeals or tropes, an especially suitable exercise to introduce and practice coding latent variables.

Step One: Articulate Research Questions/Objectives

Advertising is a goal-driven activity and advertising research is no different. Articulating well-formed research questions/objectives is essential. For the purposes of introducing content analysis as a research methodology, first exploring within class discussion the various applications content analysis has for the advertising discipline is a good start. For example: what is the competition doing? What are some recent trends (perhaps, across brands/industries) that should be noted and accounted for? What is working for successful brands? What successful ad campaigns can be studied and compared for further study?

Three content topologies useful when introducing content analysis to undergraduate adverting students are ad structure, ad strategy and appeals, and video ad execution techniques.

Table 2 provides some sample research questions, drawing on these topologies to prompt discussion and exploration within class exercise teams.

OPEN IN VIEWER

Table 2.

Sample Research Questions for Classroom Activity (Consumer Automotive—30 Second Video Ad).

ISO Product Category	Brand	Sample Questions
□ Compact □ Midsize □ Full size □ Mini Van □ Compact- SUV □ SUV	□ Chevrolet □ Ford □ Honda □ Hyundai □ Mazda □ Nissan □ Subaru □ Toyota □ Volkswagen	• What are the main structural characteristics of the ad for the product category and/or brand? • What executional techniques are typically used for the product category and/or brand? • What message strategy (source/appeals) are predominantly used for the product category and/or brand? • How does product category X compare/contrast with product category Y? • How does brand X compare/contrast with brand Y?

Important to emphasize, particularly with students who are first learning content analysis as a research method, is that research questions involve assumptions and goals about a universe of discourse (i.e., the thing being studied and analyzed). Therefore, the research questions posed in Table 2 are, in some important sense, arbitrary. However, using industry-standard delimiters, such as brand name (model/make) as well as ISO (International Organization for Standardization) product categories, allows student teams to form nuanced descriptions and comparisons. For advertising educators who use this exercise in multiple sections or successive courses, more quantitative research questions and hypotheses are also an option since prior classroom activity results can be used for benchmark comparisons and hypothesis testing.

Step Two: Prepare the Data—Create YouTube Playlist

Defining a universe of discourse for a content analysis research project is no simple affair. Important considerations about how to assemble the data set and the criteria for gathering and sampling are important considerations for establishing reliability within the research method. A common technique used among academic marketing scholars is, to begin with, a published list or ranking from industry experts (Clayton et al., 2012; Kim et al., 2019). US News and World Report, Consumer Reports, JD Power, and Forbes, for example, are traditional outlets that include ordinal lists as well as ranking criteria. While convenience sampling of automobile ads on YouTube, simply keyword searching on YouTube itself, is also an option and perhaps a more direct path to creating a data set, beginning with a pre-established ranking for the classroom activity helps to underscore the goals of objectivity and systematic data gathering.

One potential barrier to creating a YouTube playlist for the classroom exercise is that the Playlist Compiler needs to have a Google/YouTube account. Currently, it is not possible to create an anonymous YouTube playlist. After finding the first of ten (recommended) video ads to save as part of the playlist, the Playlist Compiler will need to choose a name for the playlist (there is a 150-character limit) and choose one of the three privacy settings: public, unlisted, and private. Since all content is already preexisting and public, choosing “public” is a good option and also allows other end-users to keyword search the playlist title when attempting to locate the playlist on YouTube.

One drawback of using publicly available videos on YouTube is that they sometimes go away (the video’s creator has the option and authority to remove the video from the platform). A best practice for creating any data set for research purposes is that the data set should be saved and warehoused for potential future use. Currently, YouTube provides a premium service that allows end-users to download and locally save the videos of a playlist. Although not required for the scope and goals of the current classroom learning activity, highlighting YouTube Premium features may be useful for more formal and practitioner-oriented research projects.

Step Three: Stipulate Units of Analysis—Develop Codebook

The variables selected for content analysis need to be closely aligned with each research team’s research questions/objectives. Satisfying conditions for validity and making sure each research team is “measuring what they in fact think they are measuring,” depends in large part on the accuracy and detail of how the units of analysis or variables are described and defined. Codebook definitions should aim to pick out “all and only” those things being measured or counted. Audiovisual content often involves layers of complexity, and the goal of a good codebook is to allow coders to independently recognize and record the various parts or units of an advertising message.

Table 3 includes three content topologies that are useful for analyzing video ads and which draw on familiar topics of many advertising courses: ad structure, ad strategy and appeals, and execution techniques. Sample codebook definitions are provided; however, some variables may need fuller descriptions and will be amended after teams pretest and code sample ads. It is important to emphasize at this step that defining units of analysis is an iterative process, requiring group discussion and consensus. Brainstorming how current definitions may be ambiguous or unclear is an important check for teams to consider. This step of the process may also invite new or special variables to be added and reconsidered as part of the team’s research questions. Identifying tropes within advertising relies on experience and expertise. The ability to describe and define such elements are useful skills that can lead to valuable insight. For the purpose of the classroom exercise, we recommend that no more than 15 variables be analyzed.

OPEN IN VIEWER

Table 3.

Sample ‘Units of Analysis’ for Classroom Activity (Selected For Automotive Industry Video Ads).

Content Typology	Unit of Analysis	Example Codebook Definition
Ad strategy	Comparative Appeal a	Direct or indirect reference to competition is made. Ad message differentiates the product/brand on quality, features or price by comparison with competitor.
Ad strategy	Humor Appeal a	Frames the ad’s argument or main point via a humorous or funny context.
Ad strategy	Rational Appeal a	Explicit reasons and facts are presented in the ad’s message as a means to persuade, convince, or influence the audience.
Message structure	Primacy effect a	The ad’s argument or most important point is placed at the beginning of the ad.
Message structure	Recency effect a	The ad’s argument or most important point is placed at the end of the ad.
Message structure	Conclusion drawing a	The ad includes a message that draws an explicit conclusion. This may include raising a question and providing its answer or it may involve providing supporting reasons for a stated conclusion.
Message structure	One-Sided message a	The ad states only positive points or attributes related to the product/brand.
Message structure	Two-Sided message a	Ad includes both “pros” and “cons” of the product/brand; good and bad points are mentioned in the context of the product/brand.
Execution technique	Slice of Life b	Depictions of how product can be used or experienced by the customer is provided either by life-action, animation or image montage.
Execution technique	Spokesperson b	Spokesperson for the product/brand may appear as “talent” (seen speaking on camera but not directly to the audience); as a “voice-over” (heard but not seen); or as an announcer (seen and heard and speaking directly to the audience).
Execution technique	Testimonial b	A real person, or actor portraying real a person, endorses the product/brand and/or explains that benefits that personally experienced by him/her/them.
Execution technique	Demonstration b	Demonstrates actual product/brand features and/or attributes and their perceived benefit(s).
Execution technique	Vignette b	A reoccurring character or situation to highlight the product/brand’s key consumer benefit or value. Ad is typically one of several to appear over time.

Step Four: Establish Categories and Coding Scheme

The units of analysis or variables for content analysis are typically bivariate: two categories, involving a “yes” or “no;” “present,” or “not present;” a “1” or “0.” Creating categories for a coding sheet may also include two other states of affairs: “cannot determine” (often coded as a ‘2’) and “systematic error detected” (coded as a “99”). A systematic error might be some feature of the working codebook that does not consider a feature or aspect of the data set and, unless corrected, will need to be ignored and removed from consideration. Similarly, depending on how the coding process is set up, there will be differing techniques for addressing missing and indeterminate data. In some cases, the reoccurrence of a “99” result will prompt a revision of the codebook and operational definitions.

Modern electronic forms provide a convenient way to create a coding sheet for data collection. Google Forms, SurveyMonkey, and Qualtrics, for example, provide a flexible and sharable interface that provides some “built-in” tools for data visualization. Whereas Qualtrics is perhaps considered the more advanced tool, with a steeper learning curve and a higher entry cost (i.e., for a license/subscription), it offers a broader range of features for analyzing and interpreting data. Google Forms provides a good quick-start “free” option for teams interested in visualizing their results and it also integrates well with other Google apps, such as Google Sheets and Google Slides. In addition to listing each variable in an order that is intuitive for coders, the coding sheet should provide input categories for the management and documentation of each individual ad. For example, date and time the ad was coded, by whom, name or number of the ad (as it appears in the playlist/data set), and possibly, total time required to analyze the ad and original release/airing date of the ad are useful information when processing the data. For the classroom activity we recommend with automotive ads, we suggest a minimum of 10 ads to be included in the YouTube playlist, where each ad should be (roughly) 30 seconds in length. The 30-second time constraint helps control for a common format and purpose for the ad since a standard convention for television ads is the 30-second time frame. Providing space within the coding scheme for coders to note features of the ad as metadata can also provide additional opportunities for analysis.

Step Five: Pretest, Check for Validity, and Tweak Coding Instrument

Class exercise teams should reserve time to review each variable and codebook definition that is included on the team’s code sheet. Brainstorming and discussing how issues or ambiguity may arise in the coding process is a helpful exercise. With all members present, teams should next pretest the first two ads that are included on their team’s YouTube playlist. It is advisable that each team member access the playlist on their own device (e.g., on a smartphone or personal computer) and preferably with headphones or an individual earpiece. Maintaining independence as a coder is an important standard to emphasize during the coding process. The pretest is a convenient occasion to set expectations for how each coder will evaluate the playlist/data set and only uses the codebook definitions as a guide. We recommend using the first two ads in the YouTube playlist for the pretest with the team’s agreed-upon coding sheet (developed in steps three and four). Although only two team members will code the entire playlist/data set (in step six), it is recommended that all team members participate in coding the first two ads in the pretest to facilitate discussion and overall consensus.

The pretest should conclude with a cursory check of agreement and disagreement among all coders. Discussion of any non-obvious decisions is also required. Opportunities to amend, change, or update the codebook definitions may be presented. Establishing a context for common knowledge is also possible. Team members may, for example, recount an actual example from prior experience with video ads and include those alongside the codebook definitions in the codebook Figure 2.OPEN IN VIEWER

Step Six: Code Data Set and Assess Consistency

This step of a content analysis tends to be the most time-consuming. The two members of the team assigned as coders might consider completing this step outside of class as homework. Wherever the actual coding of the YouTube playlist takes place, however, it is important to emphasize that each coder should be separate and independent in their evaluation of the data. It is worth repeating: coder independence, along with clearly specified rules and codebook definitions, are two formal requirements that help ensure objectivity and validity. If the number of items content analyzed in each video ad is kept relatively short (say 10–15 units of analysis) then the amount of time required to code the playlist is roughly similar to the time required to create the playlist (e.g., performed by the Compiler). Coders typically find success first watching the entire ad in full, one time through, and then using a “scrubbing” technique, where the YouTube playhead or cursor is manually advanced or reversed on the video’s timeline, in order to make decisions related to the team’s code sheet.

Making use of an electronic form coding sheet, such as Google Forms or Qualtrics survey/code sheet, allows for a quick and easy way to assess consistency, at least initially. Table 4 reviews three common methods to check for coder consistency. Although certainly not comprehensive for how coding consistency or intercoder reliability can be assessed, it does provide an introduction to this important area of analysis when processing and evaluating data.

OPEN IN VIEWER

Table 4.

Three Common Intercoder Reliability Measures.

Index	Formula	Description
Percentage agreement (Pa)	# o f a g r e e m e n t s # o f d e c i s i o n s	Provides a quick and easy way to calculate reliability. Does not correct for chance agreement among coders, however. Useful for detecting ambiguous codebook definitions and/or inherent latency within variables. Requires two or more coders.
Holsti’s formula ( H )	2 M N 1 + N 2 M = # of agreements b/w coders N = total # of decisions by each coder	Similar to the percentage agreement a hard return here to space things out method but allows for missing data or a different number of decisions made by each coder (Holsti, 1969). Allows for the data set to be divided-up and distributed among multiple coders (thus useful for very large data sets). Typically, indices for each variable are reported, along with the mean and range for all variables analyzed. Does not correct for chance agreement among coders a hard return here to space things out.
Cohen’s kappa ( Κ )	% o b s e r v e d a g r e e m e n t − % e x p e c t e d a g r e e m e n t 1 − % e x p e c t e d a g r e e m e n t	A measure used with only two coders. Corrects for chance agreement among coders. Requires guidelines for interpretation. Fleiss (1981) is commonly accepted for many studies in marketing and advertising, where Κ values over .75 are considered excellent; .40 – .75 fair/good; and .40 and below are poor. Calculating “expected agreement” may require constructing a “confusion matrix” and/or utilizing statistical software.

In addition to the standard “percent agreement,” considerations about chance agreement and missing data are addressed in either Holsti’s formula ( Η ) or Cohen’s kappa ( Κ ), briefly described in Table 4. As the complexity and length of media content analyzed increases, more subtle and advanced methods of coder consistency should be considered. Krippendorff (2004) provides a useful discussion for embracing more rigorous assessment methods.

Step Seven: Draw Conclusions and Report Findings

Once data from the coded variables are available and assessed for consistency, research teams can next draw conclusions and evaluate progress on their stated research questions/objectives. Rather than just reporting descriptive statistics, this process involves critical thinking skills, for example: How do results from the content analysis inform new understanding or insights? Are there any patterns or trends in the data that confirm or contradict prior findings or expectations? Do the results reflect standards of validity and reliability pursued and practiced during the classroom activity? What conclusions can be drawn and what insights are worth reporting?

Each team should have an opportunity to both summarize and assess the significance of their findings. A co-authored written report or a short 3-to-5-minute oral presentation are recommended deliverables for the classroom activity. The course instructor, deciding on one or the other, can also provide guidelines for what information to include in the findings report. Standard elements in a content analysis research report might include: (1) an account of the origins or premise of the team’s research questions/objectives; (2) statement of the team’s research questions/objectives; (3) characteristics of the team’s data set (i.e., what was content analyzed and what were the criteria for how it was sampled or collected); (4) examples of variables coded for, including some codebook definitions for any non-manifest units of analysis; (5) description of the process and method for how the data were coded, including the number of coders and the method used for assessing coder consistency; (6) summary of frequencies or percentages from the coded data (as appropriate); (7) key insights learned after revisiting the team’s research questions/objectives; and last (8) limitations, next steps and ideas for future research. Feedback from the instructor might take the form of a checklist, including these eight reporting elements, with the following evaluative categories: “not adequate,” “adequate,” and “exemplary.” Qualitative comments, addressing either deficiencies or things well executed in the findings report, are also recommended.