Working on the Fundamentals of Journalism and Mass Communication Research

Methodological Issues in Content Analysis

Seven articles on methodological issues in content analysis have been published in the past 10 years, examining topics such as sampling issues, computerized content analysis, reporting reliability coefficients, and the quality of data sources. For this virtual issue, we selected three articles that discuss new opportunities and challenges related to today’s new media environment.

The review essay by Lacy, Watson, Riffe, and Lovejoy (2015) responds to significant challenges in analyzing digital media content in “Internet time” (Karpf, 2012). In particular, the article discusses the validity and reliability of the “algorithmic coder” and the use of electronic databases and keyword searches. The authors argue that human subjectivity and error are also factors in Algorithmic Text Analysis (ATA), a label that they prefer to use to “capture the subjective human processes of generating the algorithm” (Lacy et al., 2015, p. 801). Sampling issues may arise because electronic databases—although often massive—might contain a nonrepresentative collection of units, and generating a complete collection of content with keyword searches depends on the terms used in the search. The authors formulate recommendations to deal with these challenges, including standards for producing valid and reliable content analysis, as well as appropriate reporting of the various content analysis steps.

Sampling issues related to electronic databases are also examined by Connolly-Ahern, Ahern, and Bortree (2009). Their study shows that (a) significantly larger samples are needed to obtain representative samples for the databases they studied (AP Newswire, Business Wire, and PR Newswire) than are required for more traditional sources of news content, and (b) constructing weeks on a quarterly basis provides more representative samples than constructing weeks based on a full year. They explain that the large number of weeks necessary to achieve representativeness is due to the unpredictable nature of, and the wide variety in, events in these databases. Their results support the idea that as information passes through more media gatekeepers, who limit and standardize content, the sample sizes required for content analysis diminish.

The article by Guo, Vargo, Pan, Ding, and Ishwar (2016) examines two “big data” text analysis methods: dictionary-based analysis and unsupervised topic modeling (i.e., Latent Dirichlet Allocation [LDA] analysis). Their study finds that LDA-based analysis performs better than the dictionary-based approach in several aspects: LDA is able to interpret more tweets, reveals more nuanced details, has greater validity, and is more cost-effective. The dictionary approach, however, remains more “focused” and might be preferred when researchers study one issue or topic, where a short list of keywords suffices. The study also shows that both methods generate significant errors.

Methodological Issues in Surveys, Interviews, and Focus Groups

JMCQ also published articles on research methodology and data collection other than content analysis. These were on ethnography, experimental methodology, media history, and two topics selected for this virtual issue: survey research, and interviews and focus groups.

Moy and Murphy (2016) discuss how shifts in technology provide opportunities and challenges for survey research today. Two developments are examined in particular. First, they consider the use of smartphones, which facilitate multimode and location-based data collection captured in the moment. Second, they look at social media, which can be used for online focus groups, for recruitment of nonprobability samples, and as an alternative to traditional survey research when intending to analyze behavioral data in social media. Given the shifts in technology and communications, the article details some best practices in the disclosure of surveys, which are critical for evaluating the quality of the figures and conclusions drawn from these data.

Rakow (2011) discusses interviews and focus groups, with the article offering recommendations for the best use of these methods, because “poorly conceived or inadequately explained research designs can leave readers muddled, misled, or skeptical of results” (p. 416). Rakow argues that researchers should use criteria such as intentionality, self-reflexivity, and transparency in decisions about the purpose, design, execution, and interpretation of the results of their projects to facilitate assessment of the value of the work. She also claims that researchers should be clear about their purpose at the beginning of the project, demonstrate that the research is intentional and systematic in its design, and provide sufficient relevant details to understand how information was gathered and demonstrate the researcher’s self-reflexivity in relation to the study.

Measurement and Scale Development

Measurement and scale development are highly relevant to JMCQ. Over the past 10 years, the journal has published 12 studies in this area. We selected two representative scale development articles and one article that focused on strategies and techniques of data analysis.

The article by Brown and Ki (2013) constructs an Organizational Crisis Responsibility Scale. The article is a fine example of a systematic scale construction process based on the recommendations of Spector (1992) and Netemeyer, Bearden, and Sharma (2003). The first steps include rigorous two-step pilot tests and a nationwide survey. The measures constructed are then further refined using exploratory and confirmatory factor analysis, resulting in a 12-item scale consisting of three items for intentionality (“the degree to which the crisis was created purposefully by a member or members of the organization”), three items for locality (“the degree to which the crisis is an internal matter”), and six items for accountability (“the degree to which the organization could have avoided the crisis”). Confirmatory factor analysis is used to test the hypothesized factor structure and confirms that the dimensions of the scale are reliable and valid.

Holbert and Grill’s (2015) article provides answers to questions you have always wanted to ask but were afraid to, and offers an extremely useful and understandable overview of structural equation modeling (SEM)-based confirmatory factor analysis procedures, which are highly relevant to measurement and scale development. The authors pay attention to issues such as testing competing models, proper model identification, the meaning and assessment of covaried error terms, the appropriate use of recommended model fit statistics, model equivalency, the comparison of single- and multiple-group models, and the introduction and testing of equality constraints. In particular, they call for the testing of competing and multiple-group models. They also emphasize the distinction between a reflective (a scale) versus formative (an index) relationship between observable and latent variables. They illustrate their argument with secondary analysis of World Values Survey data.

Choi (2016) offers a fine application of one of the recommendations of Holbert and Grill, comparing several competing confirmatory factor analysis models in developing a scale for news sharing. Based on theoretical and statistical considerations (the second-order, two-factor oblique model fitted the data best), the author concludes that news sharing is based on two distinctive behaviors: news internalizing (by those who read the news) and news externalizing (by those who offer news to others). News browsing and personalizing are two subdimensions of news internalizing, whereas news recontextualizing and endorsing are subdimensions of news externalizing.

The Road Ahead

Theoretical developments and technological innovations will continue to give rise to new demands, possibilities, and challenges in relation to data collection, measurement, and analysis. Journalism and mass communication data sources have become fully digitalized in recent years, and their volume, variety, and the speed of updates continue to drastically increase. The field now encompasses not only traditional online news sites, but also social media, organizational websites, forums, blogs, and so forth, as well as all the interactions that users have with these texts, and within their own social networks. Moreover, new topics are continually emerging, such as the role of fake news, filter bubbles, personalization, and even communication between humans and conversational agents enabled by artificial intelligence (such as chatbots, virtual assistants, or recommendation systems). Both quantitative and qualitative approaches are needed to study these topics, and will change as a consequence of these shifts in technology. It can be expected that algorithmic approaches and computational methods will gain an even more prominent role on research agendas in the near future, both as objects and as methods of study. Subjects on the agenda will include advances in ATA aided by ever more sophisticated natural language processing techniques, machine vision as an alternative for coding increasingly larger volumes of visual content, and other implementations of supervised and unsupervised machine learning and topic modeling. As digital information is often personally identifiable, ethical questions and privacy issues will become increasingly relevant. It is obvious that these issues of method will require the multidisciplinary collaboration of journalism and communication scholars with colleagues from information science, law, and other relevant disciplines.