Thematic Proximity in Content Analysis

Observation and Theory-Based Coding

The purpose of explorative (inductive) content analysis is to identify analyzable units (codes) in primary or secondary text data (newspapers, office documents, interview transcripts, field notes, etc.) and to summaries them under meaningful labels (categories). Depending on how complex the material under investigation is, the researcher has to decide how to organize the units of analysis. There are at least two approaches to this: According to the Coding Manual for Qualitative Researchers (Saldana, 2013) on the first level, the researcher attaches a code to certain segments of text. On the second level, he organizes interrelated codes into categories thereby creating a taxonomy or category scheme with different categories and subcategories. On the third level, the researcher arranges groups of categories into themes. On top of this pyramid stands a “theory” about the subject as the result of the analysis.

Other approaches allow for coding themes directly into the data, without the process of extracting codes and building categories. This is called thematic coding (or “themeing” the data according to Saldana, 2013, p. 175).

According to one common practice of thematic coding (Attride-Stirling, 2001), one can use three hierarchical levels, or category classes named basic, organizing, and global themes to discriminate in different degrees between rather abstract and rather concrete content. A basic theme is “the most basic or lowest-order theme that is derived from the textual data,” an organizing theme is “a middle-order theme that organizes the Basic Themes into clusters of similar issues,” and global themes “are super-ordinate themes that encompass the principal metaphors in the data as a whole” (Attride-Stirling, 2001, p. 388).

So the main point of difference between the two approaches is whether the researcher can apply (or identify) an analytical unit from the third level directly to data. The common ground of both approaches, and this is the decisive point of the methodology proposed here, is that these analytical units have a hierarchical order.

Next to this organizational structure, the coding procedure also requires coding heuristics: standardized rules that guide the decision of the researcher about when to create a new analytical unit, how to label it, and how to separate codes, categories, and themes from each other. It is important to spell out rules and thereby make the coding and classification process as transparent and replicable as possible. For calculating the theme-correlation coefficient, use the following heuristics (Heuristics 1-3; taken from Kelle & Kluge, 2010).

In explorative content analyses, the researcher usually has to code the entire data set, or at least substantial parts of it, several times until the coding scheme becomes stable. During these iterations, the coder creates, modifies, deletes, and merges the units in accordance to the coding heuristics, thereby steadily developing the taxonomy.

When conducting theory-based (deductive) coding, the researcher starts with a given set of analytical units (codes, categories themes), that is, the number and the label of units is fixed through the theory from which they derive. Coding Heuristic 1 therefore does not apply to theory-based coding, but Heuristics 2 to 4 do. If internal and external homogeneity cannot be achieved, then this indicates a mismatch between the theory and the data.

Thematic Proximity

Coding is a time-consuming and tedious process and of course not an end in itself. One purpose of coding is to reduce complex text structures to analyzable units. A fully coded set of documents enables the researcher to address a wide range of research questions with a large repertoire of (qualitative and quantitative) analytical approaches. One of them is the analysis of thematic proximity (the relation between units). Its purpose is to identify latent patterns in the content that cannot be observed by simply reading the material. The analysis of latent patterns is called relational content analysis.

Relational content analysis usually combines qualitative and statistical interpretation of verbal data into one coherent instrument (Bos & Tarnai, 1999). Still it is not a mixed methodology in the strict sense of the term, insofar as it does not necessarily require collecting “both quantitative and qualitative data” (Creswell & Plano Clark, 2011, p. 276; Onwuegbuzie & Teddlie, 2003). The “mix” is the stage of analysis where a numeric coefficient indicates how strong two themes are related to each other. Still, the detection and interpretation of meta-themes goes beyond “quantitative analysis of qualitative information” (Fakis, Hilliam, Stoneley, & Townend, 2014), and is more than just “two separate approaches to studying the same phenomena” (Symonds & Gorad, 2008, p. 11).

There are different means available to determine the thematic proximity of two descriptive units. The work of Oleinik (2011) provides a useful overview. Cosine similarity, for example, is a vector-based method often used in automated text mining, such as Leximancer.

To a certain extent, theme relation coefficients resemble metrics for intercoder reliability, such as Krippendorff’s alpha (Krippendorff, 1995, 2004; Neuendorf, 2017). Both indicate code overlapping, however, for different purposes. It could be worthwhile to “hack” alpha coefficients in a way that they indicate thematic proximity instead of intercoder reliability.

Another means to determine thematic proximity is to analyze the frequency and pattern of theme co-occurrences within a given set of documents. The c-coefficient used in content analysis software ATLAS.ti (Friese, 2014) measures how often and consistently two codes co-occur or overlap throughout the entire text sample. The c-coefficient represents the frequencies and patterns of code co-occurrence “similar to a correlation coefficient statistics” (Friese, 2014, p. 189). It is based on the Jaccard similarity coefficient. ³ Many content analysis software packages do not provide this function, but the researcher can use the code retrieval function “near within one paragraph” to determine the number of co-occurrences, and then calculate the c-coefficient manually using the formula (Friese, 2014, p. 190):

c = n 12 ( n 1 + n 2 ) − n 12 ,

with n 1 = the number of occurrences of Code A, n 2 = the number of occurrences of Code B, and n 12 = the number of co-occurrences of both codes.

Co-occurrence means that that both codes either code the same segment or overlapping segments. The coefficient can take values between 0 (indicating perfect independence) and 1 (indicating perfect relation). The greater the discrepancy between n 1 and n 2 , the smaller are the highest possible values of c. For example, if the number of code occurrence of Code A are twice as high as those of Code B ( n 1 = 2 × n 2 ), the maximum value of c is 0.5 indicating that Code B occurs always in combination with Code A, whereas Code A occurs in 50% of occurrences together with Code B.

The c-coefficient has two important limitations: First, it can underestimate the strength of a thematic relation when one analytical unit has considerable more codings (the number of discrete text segments that are associated with a given code) than the other. ⁴ The c-coefficient does not take into consideration the proportion of overlapping content. Therefore, it can remain low although the thematic link might be quite elaborated in terms of word frequencies. Second, it is not standardized and disregards the overall coding pattern of the data set making it difficult to compare c-coefficients from different studies.

To prevent this loss of information, the following section proposes an additional coefficient that takes into account not the frequency of code co-occurrences but the proportions of text intersections based on word frequencies. Taken together, these two coefficients can better assess the qualitative and quantitative relation of two themes.

Standardizing the Theme Relation Coefficient

It is important to note that the sample size, coding heuristics, and number of descriptive units can affect the t-coefficient. What may be a high coefficient within the scope of one study may indicate a rather weak thematic relation within the other. This obstructs comparability of the t-coefficient between two studies. To eliminate the influence of the coding practice on the results, it is necessary to calculate the standardized t-coefficient t s . The standardized t-coefficient eliminates the influence stemming from different coding practices, namely, the overall proportion of multiply coded content, and the number of units in the coding scheme. The more content is coded by multiple units and the fewer the number of units in the coding scheme are, the higher are the average values of the t-coefficient (and vice versa). This is due to the fact that the unstandardized t-coefficient is based on the proportion of intersecting words between units. To compare the coefficient between different content analysis studies, it is therefore necessary to consider the net effect caused by coding practices.

The standardized t-coefficient is adjusted in regard to these two general coding patterns. It is calculated in four steps. First is to calculate the proportion of text retrievals with more than one coding in relation to the sum of all text retrievals:

P = p r p s ,

where p r = the word frequency of text retrievals with more than one code in the entire sample, ⁶ and p s = the word frequency of all retrieved text segments. ⁷

P states the extent of multiply coded text in all documents of the sample. The P value in the showcase study is P = .62 and states that 62% of all coded words are coded with more than one unit. ⁸ P has to be interpreted in relation to the degrees of freedom, that is, the number of all possible (yet, not measured) bivariate correlations. The higher the number of categories within the coding scheme, and with that, the number of possible bivariate correlations, the lower is the average influence of P on any given bivariate correlation.

The degrees of freedom are determined by the number of potential correlations between the k categories. They are calculated by dividing all fields of the code-correlation matrix ( k 2 ) minus the fields in the diagonal by two.

d f = ( k 2 − k ) 2 .

The next step is to calculate the adjustment coefficient U. It indicates the average bivariate correlation if all multiply coded text were equally distributed among all units of the category scheme. It works as a baseline comparison for the observed correlation t.

The adjustment coefficient U is U = P / d f .

The standardized t-coefficient is: t s = ( 1 − U ) t .

It is in the judgment of the researcher to decide whether to report every single standardized t-coefficient. In some cases, it might be sufficient just to report the overall adjustment coefficient U, namely, when U is so small that it hardly affects the difference between t and t s .

Interpreting t-Coefficients

There are two ways to judge whether a given t-value indicates a weak, moderate, or a strong relation. First is to compare different t-values with each other. As can be seen in Table 1, the highest correlation between two categories is .129 between the theme “theological justifications for the use of force” and the narrative about the “global conflict.” Compared with other thematic links, this is strong. The c-coefficient (.058) is also comparably high signaling that this link is also more frequent and consistent than most other links in the table.

OPEN IN VIEWER

Table 1.

Theme Relations.

Themes issues	Diagnostic frame			Reference system
	Apostasy (39,128)	Global (20,730)	Secular (10,428)	Factual evidence (46,641)	Theological evidence(21,832)
Instrumentality of force (7,657)
Strategic benefits (4,973)
Intersection	533	488	187	1,131	137
c-coefficient	.036	.045	.048	.031	.009
t-coefficient	.060	.061	.028	.126	.017
Religious benefits (2,684)
Intersection	603	82	283	312	71
c-coefficient	.018	.005	.017	.006	.026
t-coefficient	.120	.017	.066	.061	.015
Justification for the use of force (13,075)
Political justifications (3,010)
Intersection	15	518	69	132	51
c-coefficient	.011	.066	.016	.024	.003
t-coefficient	.003	.099	.015	.023	.010
Theological justifications (10,065)
Intersection	1,925	1,753	1,61	1,033	1,182
c-coefficient	.050	.058	.023	.016	.110
t-coefficient	.120	.129	.016	.062	.086

Another way to judge the strength of the correlation is to compare observed t-values against the unobserved t-values of two mutually independent themes (here U = .0002). A t-value of .129 then indicates that the correlation is significantly different from independence. It is also possible to base this benchmark test on randomly, instead of equally distributed content. This would in some way resemble the statistical test for significance and could be the method of choice in quantitative content analysis.

The standardized coefficient t s is always smaller than the observed coefficient t, but it has the advantage that it is not affected by coding practices, such as the number of units in the category scheme, and therefore is more suitable to compare results.

Detecting Meta-Themes, or How to “Read Between the Lines” of Qualitative Data

When two or more descriptive units systematically co-occur in the text data and when the co-occurrence is not only frequent but also elaborated in terms of word frequencies, then this can indicate the presence of a meta-theme. The two coefficients therefore are quantitative indicators for meta-themes. Meta-themes are themes which acquire their meaning through the systematic co-occurrence of two or more other themes. The prefix “meta” means that these themes are themes of a higher informational order, or in other words, they are not explicitly but implicitly communicated within the content. A meta-theme might mark subconscious communication and tells the researcher something about the source, namely, that it systematically refers to two distinct themes.

The statistical coefficients should always be interpreted in combination with a qualitative assessment of the meta-content. Not every thematic correlation is necessarily a meta-theme. Likewise, the detection of a meta-theme does not necessarily reveal the reason why the originator communicates subconsciously and not explicitly and intentional. This question can be answered only within the context of a particular study.

To give one example of subconscious communication, we now turn to the showcase study.

Literature Review: Content Analysis of Jihadi Media

Over the last 15 years, the Jihadi movement has produced an abundance of media and propaganda material, and the academic community was not idle to investigate this material with a great deal of interest. Despite the wealth of available data and scholarly work, systematic content analysis of this material is still the exception. It seems that the availability of highly interesting and politically relevant research material was conductive for an atmosphere in which “the terrorism studies community seems to have deviated from the guidelines of academic conduct” (Hellmich, 2008, p. 111). The availability of primary sources coincided with the “post-9/11 money surge into terrorism studies” for which Marc Sageman (2014) provocatively diagnosed “deleterious effect” (p. 566). Although there are also examples of good scientific practice, terrorism studies have not yet exploited the full potential of content analysis approaches.

Authors of studies who apply content analysis techniques often remain descriptive. Eveslage (2013), for instance, counted the number of threats against domestic and foreign targets within 23 public statements of the Nigerian Jihadi group Boko Haram. Torres, Jordán, and Horsburgh (2006) used qualitative and quantitative thematic coding to summaries a sample of 2,878 documents from AQ. Salem, Reid, and Chen (2008) classified 706 media files produced by Jihadi groups in regard to their production features, purpose and usage as documentary, propaganda, operational, hostage, executions, statement/communique, tribute/eulogy, training, and instructional videos. Pennebaker and Chung (2008) described differences in linguistic styles between bin Laden and Zawahiri, and Beutel and Ahmad (2011) inferred from their analysis of 49 bin Laden speeches, that the now deceased leader of the Jihadi movement cited policy-based grievances for his militancy twice as often as religious-based ones.

Descriptive content analysis of Jihadi media gave researchers a first glance into the wealth of data but to come to more generic conclusions about the groups who communicate these messages, more sophisticated analysis is needed. A common approach in terrorism studies therefore is to compare extremist groups who engage in violence with those who do not (A. G. Smith, 2004). For example, A. G. Smith (2008) and A. G. Smith, Suedfeld, Conway, and Winter (2008) applied three psychological measurement constructs (value reference, motive imagery, integrative complexity) to media content of violent and nonviolent Islamist groups, and identified those variables that are statistically significant predictors to distinguish between groups. Conway, Gornick, Houck, Towgood, and Conway (2011) investigated “hidden implications of radical group rhetoric” by analyzing random text samples with integrative complexity coding from violent and nonviolent Islamist groups. Pennebaker (2011) identified in a text sample of 296 documents statistically significant predictors for a violent attack in the 2 to 6 months following the statement of the group. Rieger, Frischlich, and Bente (2013) integrated ethnographic content analysis of Jihadi and right wing media into a randomized experimental design to investigate the individual’s response to ideological messaging.

Methodology of the Showcase Study

hierarchical levels of analytical units

Using software MAXQDA, ¹⁰ I combined a theory-based coding with explorative coding into a hybrid coding design. Therefore, the coding structure includes both theoretically and empirically driven units of analysis, also referred to as deductive and inductive categories (Mayring, 2000). It has five hierarchical levels:

Ideology as discourse (theory driven)

Frame (theory driven)

Narrative (global themes)

Theme (organizing themes)

Issues (basic themes)

Basic, organizing, and global themes (Attride-Stirling, 2001) or codes, categories, and themes/concepts (Saldana, 2013) represent the empirically driven units. To utilize these units for the particular purpose of studying ideologies, I call them “issues,” “themes,” and “narratives.” They discriminate in different degrees between rather abstract and rather concrete content within the Jihadi statements (see Figure 1). Frames and discourse are theory-driven units of analysis and represent the most general characteristics of Jihadi ideology. Discourse, frame, narrative, theme, and issue represent different hierarchical levels the coding scheme. They represent the functional elements of ideologies—the mechanisms through which they frame the world—but they do not tell anything about the actual grievances, claims, positions, strategies, and visions of the movement that embraces this ideology. Each level therefore has a certain number of descriptive categories that summaries the actual meaning of the ideology and represent the substantial elements. In the sample of 31 video statements, I identified one discourse, four frames, 11 narratives, 26 themes, and 55 issues.

OPEN IN VIEWER

Figure 1.

Category classes.

The level of “discourse” is the most comprehensive and general one. In fact, all content belongs to it. Its purpose is to acknowledge that Jihadism is not mutually exclusive from other Islamist ideologies but remains in a constant discursive relation with them, and therefore can be analyzed as such, for instance, when conducting a discourse analysis of statements published by AQ vis-à-vis statements from the Islamic State or the Muslim Brotherhood. For the purpose of this article, the analytical unit “discourse” has no further function.

The level “frames” has four descriptive units borrowed from Social Movement Theory (Snow & Benford, 1988; Wilson, 1973). Social Movement Theory has an intuitive appeal for the analysis of Islamist movements and has been used for this purpose across disciplines (Lohlker, 2013; Snow & Byrd, 2007; Wiktorowicz, 2004a, 2004b). It states that all ideologies are comprised of three principal components, also called frames: The “diagnostic frame” of an ideology describes (perceived and actual) social problems (i.e., “the war on Islam”) and specifies alleged political, economic, and social reasons for these problems. The “prognostic frame” describes the goals the movement pursues, namely, to replace the unjust status quo with an auspicious alternative (i.e., “the caliphate”) and the “motivational frame” describes strategies how the goals can be achieved (e.g., “jihad”). For coding purposes, I used a fourth frame (reference frame) as an auxiliary unit to designate all content that is nongenuine, that is, when the authors of the statements refer to external sources to substantiate their socioreligious positions, claims, and grievances. For instance, Jihadi leaders use theological evidence (references to Quran and Sunnah) to substantiate their theological argumentation, factual evidence (references to mainstream media or governmental reports) to back up their political claims, and aesthetic “evidence” (Islamic poems and lyric) to increase the “narrative fidelity” (Snow & Benford, 1988, p. 210) of their message.

When conducting hybrid coding, one can start the coding procedure top-down by coding the most general units into the data, or bottom-up by looking for the smallest informational units first. Starting with the most general (theoretically driven) unit has the advantage that it usually requires little prior knowledge about the content. It also gives the coder a first glance into the material so that he gets a rough idea about the thematic complexity and the approximate number of empirically driven (inductive) themes present in the material. In the study of Jihadi media, it was straightforward to recognize whether the author of the statement describes the status quo, talks about his vision or utopia, or advices followers to take action. In the most simplistic manner, coding frames into ideological statements follows the ABC model (Account, Better World, Change) of Mark Sedgwick (2012). Unlike the empirically driven units, frames must be mutually exclusive. However, the empirically based subunit of frames can cut across two or even three frames.

The next task is to identify the empirically driven themes. Here the researcher starts from the scratch with nothing else than the four coding heuristics (see above) to guide him. Processing one statement after the other in no specific order, I created, modified, deleted, and merged the descriptive units in accordance to the coding heuristics, thereby steadily developing the coding structure. After working through 10 statements, the coding structure began to stabilize, meaning that fewer new units emerged and that fewer modifications were necessary to satisfy the coding heuristics. At the end of the first coding iteration, the coding scheme was entirely stable and the last few documents did not trigger any more modifications. This indicates that the coding structure represents the content adequately and also that the sample is saturated. A second coding iteration was necessary to adjust the content of the first processed documents to the finally developed scheme. The final version of the scheme has four frames, 11 narratives, 26 themes, and 55 issues. To visualize the thematic structure, I created a mind map that depicts all 96 categories (Armborst, 2013).

Results: Interpreting t-Coefficients and Detecting Meta-Themes in Jihadi Media

The systematic content analysis approach has helped to clarify and dissect the otherwise rather indistinct bulk of ideological messages. The main research objective of the study was to explore the ideological origins of religiously inspired violence in Jihadism. The analysis shows that Jihadism is a complex ideology that touches on a plethora of explicit socioreligious issues. The main thematic structure of the ideology consists of four frames, 11 narratives, 26 themes, and 55 issues. It contains rigorous theological argumentation mixed with political analysis expressed in the language of journalism or even scholarly argument. It is beyond the purpose of this article to describe all these aspects in detail. The important point here is to show the application of the theme relation coefficient.

Figure 2 and Table 1 present some of the quantitative results of the showcase study. Within the motivational frame of AQ ideology, two narratives and four themes are of particular interest in regard to the research objectives: the narrative about the (1) “instrumentality of force” in which the authors describe what they think the movement can actually achieve through the use of force. These expectations are further detailed within the two themes (1.1) “strategic benefits” and (1.2) “religious benefits.” The other narrative is the (2) “justification for the use of force” with its two themes: (2.1) “political justifications” and (2.2) “theological justifications.”

OPEN IN VIEWER

Figure 2.

Text proportion for frames, narratives, and themes.

To operationalize the broader research objective, I formulated the following working question: Which other narratives, themes, and issues co-occur systematically with (1) and (2), and how strong are the thematic relations between them in terms of quantity and quality? Relational analysis helps to assess how the rationale of violence is embedded in the wider narrative structure of AQ’s ideology, not only in terms of statistical co-occurrence but also in terms of elaboration and meaning.

Figure 2 shows the absolute and relative word frequencies of selected categories. Beginning with the most extensive narrative (about apostasy), categories are ranked and grouped according to the hierarchy of the coding structure (frames, narratives, themes). The information about word frequencies helps to put the qualitative description of each frame, narrative, and theme into a broader perspective about the general outline and composition of Jihadi ideology. It empirically supports the observation made in other studies that Jihadism is mainly about Islamic rivalry (the near enemy) and to a lesser degree concerned with geopolitical affairs (the far enemy), but both aspects are certainly connected, as the relational analysis shows.

The coefficients in Table 1 reveal how frequent and how strong categories are linked. It displays c- and t-coefficients for the thematic relation between the four themes about the rationale of violence (rows) and the three narratives within the diagnostic frame (columns). The numbers in the table can be interpreted in a similar way than a crosstab with categorical variables. To give a reading example of the numbers in the table, the narrative about the instrumentality of force (7,657 words) has two subthemes: strategic benefits (4,973 words) and religious benefits (2,684 words). These subthemes correlate to different degrees with the three narratives in the diagnostic frame (apostasy, global conflict, and secular governance) and are backed up to different degrees by factual and theological evidence from the reference system (nongenuine content). To pick out one example, the two categories “strategic benefits” and “apostasy” share 533 words, which correspond to a c-coefficient of .036 and a t-coefficient of .060, indicating a moderately elaborated and rather infrequent thematic relation.

As noted before, the coefficients should always be interpreted in connection with a qualitative assessment of the thematic link. When reviewing the meta-content (533 words) cutting across the themes “strategic benefits” and the “apostasy” narrative, it reveals a tacit message: AQ asserts quite plainly that jihad is as much a matter of strategic choice as it is a matter of Islamic law and individual duty. What they claim rather implicitly is that this distinction makes them superior to competing Islamist groups who act much more strategically (“opportunistic” in the view of AQ). AQ promotes active participation in jihad, even against all strategic odds, to demonstrate its pristine interpretation of Islam and to claim religious supremacy over competing Islamist movements (often labeled apostates) who refrain from the alleged duty of jihad for purely political and strategic considerations.

But why is this claim communicated implicitly rather than directly? A plausible explanation is that the strategic flaws of Jihadi military doctrine are both a powerful and vulnerable aspect of the ideological message. It is powerful because only this way AQ can credibly claim religious supremacy over competing Islamist groups. And it is a vulnerable point because AQ can be (and actually is) criticized for being strategically and militarily ineffective and therefore not worth of support. Therefore, this aspect of the ideology has to be communicated in a subtle way as to disguise its contradiction.

This observation is an intriguing and important aspect of AQ’s ideology, much more important than the rather moderate correlation of t = .060 would suggest. This demonstrates that it is important to review the statistical results always in combination with a qualitative assessment.

Other themes systematically co-occur without carrying any implicit message. For example, the theme about the strategic benefits of jihad are backed up quite strongly (t = .126) by factual evidence but not so strongly by theological references to Quran and Sunnah (t = .017). There is nothing more to conclude from this observation other than the Jihadists use rational (factual) instead of theological reasoning when describing the strategic utility of Jihadi warfare.

Finally I want to use the showcase study to give an example of how to interpret the unstandardized coefficient t together with the standardized coefficient t s and how both are affected through the coding pattern. The adjustment coefficient in this study is almost 0 (p = 62/df = 4,560) indicating that there are few code overlaps in relation to the overall number of units. As much as 62% of all retrieved content ( P s ) is coded with more than one code ( P r ) , but there are also many degrees of freedom (categories among which the multiply coded text can freely distribute). Between the 96 units of the coding scheme, there are df = 4,560 possible (though not observed) bivariate correlations to accommodate the P r = 132.732 words that appear multiple times in the text retrieval. If all 96 units were perfectly independent from each other (in other words, if all multiply coded text were equally distributed among all 96 units), then any bivariate correlation would be close to .001 indicating that an observed coefficient t, for example t = 0.1, is significantly higher than the average correlation between two units. Therefore, the standardized coefficients t s = 0,9998 × t will take almost the same values as t and must not be reported.