Abstract
In this article we present a content analysis framework for textual analysis of programmatic documents with the goal of identifying party positions on the ethnic dimension of political competition. The proposed approach allows for evaluation and comparison of how party systems in multi-ethnic states process ethno-cultural claims and demands. Our method of content analysis of party programmatic texts provides adequate granularity by which to capture the subtleties of ethno-cultural political rhetoric. It also addresses some of the misclassification and measurement problems raised in the literature with respect to the dominant Comparative Manifesto Project (CMP) approach to textual analysis. We demonstrate how estimates generated by our method for human-based coding constitute an improvement on the CMP’s estimates of party positions on ethno-cultural issues.
Keywords
Introduction
Electoral manifestos of political parties have been used for estimating party positions in a large number of policy areas. The ethno-cultural dimension of party competition is not one of these areas. Having estimates of parties' positions on ethnic issues, along with estimates on other policy issues, is important both for empirical description and theoretical model-building. Such estimates help to operationalize concepts that are at the core of literature dealing with the formation and persistence of group identities, ethnic mobilization and nationalism, diversity management and power-sharing (Chandra, 2004; Gurr, 1993; Hechter, 2000; Horowitz, 1985; Olzak, 2006). These estimates are also of relevance to the general discussion on issues of party system formation and party competition in culturally heterogeneous societies (Alonso and Ruiz-Rufino, 2007; Birnir, 2007; Coakley, 2008; Lijphart, 1977). To our knowledge, there have been no systematic attempts to analyse cross-nationally whether and how political parties formulate their positions on ethnicity-related issues in party manifestos. This article is an attempt to fill this gap. It seeks to contribute substantively and methodologically to the research agenda on party manifestos.
Rather than using available methods for producing manifesto-based estimates of party positions on ethno-cultural issues, we propose a new classification scheme for human coder-based analysis. We build our approach on research conducted by the Comparative Manifesto Project (CMP), which is by far the most comprehensive effort to provide a comparative framework for the analysis of party manifestos (Budge et al., 2001; Klingemann et al., 2006). The CMP also contains estimates of party positions on ethno-cultural issues. It is our dissatisfaction with the reliability and validity of CMP’s measures of ethno-cultural positions that provided an impetus for our effort.
While the CMP’s approach was motivated by the desire to provide a comprehensive analytical framework for capturing party positions on all dimensions of party competition, our approach is driven by the concerns of scholars interested in specific policy dimensions. Scholars studying substantive issue areas – be it ethno-cultural issues, multi-level governance or some other policy area – often find that the CMP coding scheme lacks adequate granularity to capture the subtleties of party rhetoric in that particular policy area. As the authors of the CMP scheme acknowledge, their method might be inappropriate for those who want to analyse specific policy areas (Budge, 2001b). With our study, we hope to contribute to a general discussion about the strengths and limitations of different measures of party positioning (Electoral Studies, 2007; Laver, 2001). A continuing debate on the validity and reliability of the CMP’s measures is part of this discussion (Franzmann and Kaiser, 2006; Hansen, 2008; Pelizzo, 2003).
We argue that for many research purposes there is no substitute to re-analysing party manifesto texts from the perspective of a specific policy area. The cost of the CMP’s intention to guarantee comprehensiveness of its approach is that policy positions for particular policy domains are sometimes hard to estimate using CMP data. Scholars who are disappointed with not obtaining valid results in generating specific policy positions using CMP might benefit from introducing additional coding procedures. Such procedures could be especially useful in dealing with policy areas that are represented by a small number of coding categories in the CMP scheme.
We propose a classification framework and a coding scheme for analysis of one such area. We demonstrate how our coding scheme helps to generate estimates of party positions on ethno-cultural issues which are superior to estimates produced by the CMP. For the empirical part of our investigation we focus on new democracies in Eastern Europe. Ethnic issues are generally assumed in the literature to be important for structuring the political process in the culturally heterogeneous societies of the region (Benoit and Laver, 2007; Offe, 1991). The growing awareness of the potential for ethnic differences to serve as an important source of political contestation in post-communist Europe is also reflected in the CMP’s evolving approach to coding party manifestos. A number of new sub-categories introduced into the CMP’s coding scheme in the 1990s explicitly target ethnicity-related issues (Klingemann et al., 2006).
We selected party manifestos from four Eastern European countries for our empirical investigation. These country cases – Bulgaria, Moldova, Romania and Ukraine – are multi-ethnic polities that reflect the variety of structural conditions and types of ethnic relations found in Eastern Europe. The sample captures some of the variation in salient features of these contexts; it includes countries with an established record of statehood and without prior experience of independence, with electorally successful and unsuccessful ethnic parties and with different levels of integration into European institutions. For each of these four countries, the CMP project generated detailed estimates of party positions, which enabled us to compare our results and those of the CMP. We re-analyse and present our findings on 75 party manifestos from these countries. The CMP’s original estimates are reported in its most recent volume (Klingemann et al., 2006). 1
The article is organized as follows. The first section specifies what a content analysis framework for studying ethno-cultural competition should try to accomplish. The second summarizes our arguments of why it is necessary to have an algorithm for generating estimates of party positions on ethno-cultural issues that is different from the one used by the CMP. The third section provides details of our classification framework, coding scheme and coding procedures. The fourth section applies the proposed method to the analysis of actual manifesto texts and compares our estimates and those of the CMP. In the final section, we report the results of reliability and external validity tests and conclude by summarizing the strengths and limitations of our proposed approach.
Elements of a conceptual framework for analysing ethno-cultural competition
A conceptual framework and coding scheme for analysing party competition on ethno-cultural issues should satisfy a number of criteria. Specifically, such a framework should allow specification of the content and boundaries of the ethno-cultural domain, define different policy dimensions inside this domain and identify the range of positions that parties can take on each of the dimensions.
We conceptualize party-positioning on ethno-cultural issues as a distinct domain of party competition. The existence of such a domain is often implied in ethnic politics literature (Alonso and Ruiz-Rufino, 2007; Chandra, 2005; Coakley, 2008). Party manifesto statements are defined in our research project as belonging to the ethno-cultural domain if they explicitly refer to ethnic groups and policies that affect the characteristics and conditions of these groups. Ethnic groups are understood here as communities that share (or are commonly defined by others as sharing) a set of beliefs of common descent and common culture which differentiate them from the rest of the population of a given state. This definition allows for the operationalization of boundaries of the ethno-cultural domain and for differentiating between statements that fall inside and outside this domain.
We identify two distinct dimensions in this policy domain. The first is the multicultural dimension, defined by a multicultural/integrationist divide. This divide animates many of the philosophical and policy debates on ethno-cultural issues (Barry, 2001; Kymlicka, 1996). The multicultural dimension captures party responses to the challenges posed by the ethnic diversity of societies where these parties operate. The multicultural/integrationist policy space registers party preferences with regard to the accommodation of minority groups. The variety of positions parties take and the nuances of their stances on minority accommodation issues constitute the very essence of ethnic politics. We discuss details of how the multicultural dimension has been operationalized in the party manifesto literature to date, and how we think it could be improved in the subsequent sections of the article.
The second dimension we identify is the titular ethnic group one. Unlike the multicultural dimension, which captures party differences with respect to how to deal with minority groups, this dimension is about parties staking their positions and highlighting their differences in relation to the status of a titular ethnic group. The concept of a titular group denotes an understanding of a nation in ethno-cultural terms. This understanding has been especially prevalent in the East European context (Brubaker, 1996a). The status of the titular group and the extent to which its identity and defining characteristics should be a matter for policy intervention is a highly salient issue in individual countries of the region (Brubaker, 1996b).
The two dimensions identified above do not logically overlap. A party may be running on a platform of strengthening titular group identity, but this does not necessarily predict its position on the multiculturalism dimension, on which the party can either support policies sustaining the identity of minority groups or oppose them. We mention empirical examples of such distinct bundles of policies adopted by individual parties later in the text.
Coding ethno-cultural statements: The CMP approach and its limitations
Our attempt to estimate party positions on ethno-cultural issues from party manifestos is not the first one. The CMP’s coding scheme contains a number of variables intended to capture party positions on these types of issue. We discuss the design of the CMP’s coding scheme and its limitations in some detail here because such a discussion facilitates an understanding of our method.
The CMP’s content analysis methodology is based on a classification scheme with fixed general categories that are used to cover the total content of electoral manifestos by identifying the statements of preference expressed in party texts. The coding procedure comprises a classification and quantification of manifesto statements. The coding unit is the ‘quasi-sentence’, which is defined as the verbal expression of one political idea or issue. 2 The frequency of quasi-sentences under the same category is used as an indicator of how salient a given policy issue is for the party. Central for our discussion is the CMP’s decision rule that allows for individual statements operationalized as quasi-sentences to be coded under only one of the 56 categories of the CMP’s standard coding frame.
CMP categories designed for classifying ethno-cultural statements can be conceptualized as falling into two broad types. Variables that try to capture parties' positive attitudes towards the preservation of ethno-cultural diversity can be seen as dealing with a multicultural position on ethnic issues. CMP variables that identify and code multicultural statements are: (1) ‘607 – multiculturalism: positive’; (2) ‘6071 – cultural autonomy: positive’; (3) ‘6072 – multiculturalism pro-Roma: positive’; (4) ‘7051 – minorities inland: positive’; (5) ‘7052 – minorities abroad: positive’.
Statements that signal a party’s stand against multiculturalism are integrationist. The essence of these statements is support of policies aimed at fostering a common cultural identity and at erasing or making less visible the boundaries between ethnic groups in a state. CMP variables used for coding integrationist statements are: (1) ‘608 – multiculturalism: negative’; (2) ‘6081 – multiculturalism pro-Roma: negative’.
As this summary description of the CMP’s approach suggests, the project team identified the central theme in discussions of ethnic politics – the multiculturalist/integrationist divide. This is one of the two themes we introduced in the previous section. The other theme which we argue is salient in the literature on ethnopolitics – politicization of titular group identity – is much more elusive in CMP’s approach. A number of categories, especially ‘601 – National way of life: positive’ and ‘602 – National way of life: negative’, host statements dealing with titular group identity, but definitions of such relevant categories are, as a rule, too inclusive for them to be considered valid measures of party positions on issues related to titular group identity. Therefore, we focus here primarily on examining CMP results for the multicultural dimension.
Although the CMP’s coding scheme has a number of variables that specifically target multicultural statements, this scheme is not adequate for identifying all statements related to the multiculturalism theme. A significant number of ethnicity-related statements end up being classified under the CMP’s non-ethnic variables. We label this issue as a problem of undercounting and relate it to the manifesto literature’s discussion of misclassification and measurement errors. The second major problem concerns the insensitivity of existing CMP variables to the relative importance of individual statements dealing with the same subject-matter. We label this problem as one of non-differentiation. Next, we discuss each of these issues in some detail and illustrate our discussion with examples of actual CMP data for East European party systems.
Undercounting
There are significantly more ethnicity-related statements in party manifestos than the combined frequency count of all statements coded under ethnicity-designated CMP variables would suggest. The problem is due to the fact that many statements on ethno-cultural issues are coded under variables other than those intended by the CMP to capture ethnic statements. The CMP’s classification is based on a large number of mutually exclusive and exhaustive categories, with a number of categories competing for the same statement. The coders are forced to choose between more than one plausible variable for many statements and a statement ends up in a category deemed by a coder as the most relevant. The CMP provides detailed guidelines on how these decisions are made, but these guidelines are designed for creating an all-purpose classification and not for addressing the needs of a researcher interested in analysing party manifestos from the perspective of a particular issue dimension.
A factor contributing to undercounting is the CMP’s specific decision rules that are activated when more than one coding category seems to apply to a statement. One of these rules deals with statements that might be coded either as a policy statement or a statement dealing with a social group. It instructs a coder to classify such a statement under the policy category. Out of the small number of CMP ethnicity-related categories we listed earlier, two recently introduced categories (7051 – minorities inland: positive; 7052 – minorities abroad: positive) are explicitly group and not policy-oriented. Thus, both the peripheral status of ethnic issues in the overall coding design and the ‘policy position beats group politics' rule discourage manifesto coders from using ethnic categories for classifying manifesto statements.
The problem identified above – undercounting caused by a coding procedure that forces a coder to choose between more than one plausible variable for a statement – is illustrated in Table 1 listing examples of statements that were undercounted. Examples come from party manifestos from each of the four countries included in this study. The table lists what we consider to be ethnicity-related statements which were coded by the CMP under various non-ethnic variables. As a contrast, for each of the listed variables we also provide an example of a CMP-coded statement which is not ethnicity-relevant.
Ethnicity-relevant and ethnicity-irrelevant statements in selected CMP variables.
Note: 1-ethnicity-related statement; 2-non-related statement.
Statements listed in Table 1 were coded by the CMP under one of the four categories of its standard coding frame. These are ‘202 – democracy: positive’, ‘301 – decentralization: positive’, ‘503 – social justice: positive’, ‘601– national way of life: positive’. In our view, the first of the two statements in each category explicitly provides information that is relevant for identifying the position of a party on ethnicity-related matters. Explicitly ethnicity-relevant content is what differentiates these statements from the statements listed second in each of the categories.
Given CMP’s coding scheme, all statements listed in the table were correctly attributed to their respective categories. Neither of these categories belongs to the CMP’s set of ethnicity-designated variables mentioned earlier in the text. It is our specific interest in ethnic politics that makes us view statements, which in the CMP’s classification belong to the same category, as being different. Therefore, relying on the CMP’s ethnicity-designated categories does not provide complete and accurate information on party-positioning on ethnic issues.
Non-differentiation
Another set of problems regarding the CMP’s treatment of ethnic issues deals with the fact that information obtained through the CMP’s ethnicity-designated variables is not sufficiently differentiated. With the CMP’s codes it is not possible to distinguish between different types of pledges and positions that parties take on ethnic issues. This problem stems from the same initial choices the CMP made while constructing the basic coding framework. As already discussed, the peripheral status of ethnic issues in the framework that was eventually adopted left an analyst interested in ethnic politics with only two basic categories for relevant party statements – ‘multiculturalism: positive’ and ‘multiculturalism: negative’. The other five CMP ethnicity-relevant categories provide only minor variation on these two variables.
Conceptual problems arise from other aspects of dealing with the task of classifying statements on the multiculturalism dimension. Global proliferation of human rights concerns has made political parties aware of the need to take a stand on issues of ethnic non-discrimination and protection of basic minority rights. These are increasingly perceived as part of the general body of human rights. Party statements related to these issues are treated by the CMP as falling under the ‘multiculturalism: positive’ variable. These types of statement, however, are not part of multiculturalism when the latter is understood as a positive agenda for fostering group identities and promoting ethnic diversity. They do not fall under the ‘multiculturalism: negative’ category either. The same problem exists for statements expressing general support for inter-ethnic peace and harmony. By only stating its support for inter-ethnic peace, a party says nothing about its preferences for either multicultural or integrationist ways of achieving it. By classifying all these statements as positive multicultural, the CMP’s coders are stretching the conceptual limits of the category.
Another issue is the aggregate nature of the basic categories on the multiculturalism dimension. Being the only original category for classifying statements that favourably mention or advocate ethnic diversity, the ‘multiculturalism: positive’ variable lumps together party statements of a very different nature. The following positions are all coded under the same category: support for the maintenance of cultural institutions of different ethnic groups, support for the introduction of special forms of group-based representation, demands for territorial autonomy and claims of constituent nation status. For scholars of ethnic politics, however, these types of claim are far from being equivalent (Gurr, 1993; Ishiyama and Breming, 1998).
Introducing a coding scheme for ethnicity-related statements
The issues discussed above make the CMP’s estimates of limited value for scholars of ethnic politics. Examining party manifestos through the prism of ethno-cultural competition requires a classification framework and coding scheme that is more sensitive to these scholars' interest in the systematic examination of how ethno-cultural issues are reflected in party rhetoric. We next discuss the details of our proposal for such a framework, introduce a coding scheme and discuss coding procedures.
In addressing the task of scheme development, we subscribe to the salience measurement theory that underlies the CMP’s approach to generating position scores. The salience measurement theory is contrasted in the CMP-related literature with the confrontational approach to measuring party positions (Budge, 2001a). While a ‘pure’ confrontational coding scheme would not usually use a quasi-sentence as a recording unit, the parsing of a manifesto text into quasi-sentences is the very basis of a coding scheme informed by the salience measurement theory. We retain this basic feature of the CMP’s approach – the use of quasi-sentences – in developing our coding scheme. Key differences between our method and that of the CMP relate to the coding scheme and the coding procedures that coders have to use in textual analysis.
Coding scheme and procedures
A summary of our coding scheme is presented in Appendix I. As this summary indicates, we provide a number of coding categories to identify party positions on the multicultural and titular ethnic group dimensions. While we use two of the CMP’s original category labels, ‘multiculturalism: positive’ and ‘multiculturalism: negative’, to identify the opposite ends of the multicultural dimension, we provide more restrictive definitions of what these categories actually mean. Manifested reference to the characteristics/conditions of an ethnic group(s) is a necessary attribute of a multicultural statement in our coding scheme. CMP interprets multiculturalism more broadly: some statements related to cultural practices and lifestyle choices, which are not specific to ethnic groups, are coded by the CMP as multicultural. Such statements are excluded from the multicultural categories in our coding scheme.
Our new category on this issue dimension is ‘multiculturalism: neutral’. As discussed earlier, the CMP’s dichotomous coding of multicultural positions is not discriminate enough. It does not differentiate statements which do refer to ethnicity but do not take a manifest position on whether ethnic problems should be resolved through multicultural (‘multiculturalism: positive’) or integrationist (‘multiculturalism: negative’) policies.
Table 2 provides examples of party statements that fall into each category of this revised scale.
Examples of manifesto statements classified on the multicultural/integrationist dimension.
The table should provide sufficient illustration of the types of statement we code as neutral. While acknowledging and supporting the provision of guarantees for basic minority rights, neither of the statements listed under the neutral heading advocates any positive measures aimed at fostering and promoting ethnic diversity and distinct group identities. As these examples demonstrate, having the ‘multiculturalism: neutral’ category increases the amount of differentiated information available for a researcher. It is then up to him/her whether to treat neutral statements, which are provided in our content analysis data, separately or as part of the ‘multiculturalism: positive’ category in the context of a specific investigation.
Our coding scheme also introduces a number of sub-categories for positive multicultural statements. In differentiating between multicultural statements, our point of departure is the level of demands that ethnic groups put forward in their negotiation with a state. 3 These demands can be ordered from the least to most radical ones. We differentiate four types of claim that focus, respectively, on: identity preservation, group-based political representation, territorial autonomy or group status as a constituent nation. 4 Different types of policies are usually associated with each of these types of claims and demands. Appendix I provides more details on the sub-categories of the positive multicultural statements.
We classified all statements that were identified as signalling a positive multicultural stand according to this typology. 5 Table 3 provides examples of different types of multicultural statement.
Examples of multicultural statements rank-ordered by the strength of ethnic claims.
The examples of constituent nation claims given in Table 3 might require some additional explanation. A statement about the status of the Russian language in the case of Moldova and a statement about the Hungarians' equal ownership of the state in the case of Romania were classified as constituent nation claims due to the fact that these statements directly challenge the idea of a titular or core nation. Politicians subscribing to the ‘core nation’ thesis usually support a titular group’s special entitlement claims and state policies aimed at securing a special ‘preferential’ status for a titular group’s language and culture. Statements in the last row of Table 3 question these fundamental beliefs and therefore constitute the strongest type of multicultural claim.
For the titular ethnic group dimension, we only introduce a category that extracts positive statements regarding the titular ethnic group. In the coding scheme we do not list categories for negative or neutral positions because we found them to be consistently empty. Examples of titular-group-related statements that our coders identify include the following: ‘we defend the historical right of the native population to call themselves Moldovan’ (Moldova, PDAM 1994); ‘The party supports the renaissance of the Ukrainian traditional culture’ (Ukraine, URP 1994). To reiterate our key point, the cited statements are distinct from statements we classify as belonging to the multicultural dimension. The latter focuses on ethnic minorities and the former on titular groups. The concerns about the situation of a titular group seem to be particularly widespread in the context of newly established states where majorities struggle in dealing with the legacies of their assimilation by the dominant groups of the former state.
Detailed descriptions of the classification scheme and coding categories, which are listed in Appendix I, are provided in a coding manual which has been prepared to support the use of our framework. 6 The manual also contains a description of additional decision rules to help coders make classification decisions in problematic cases when the criteria for a statement’s inclusion in the ethno-cultural domain are not fully met, the meaning of the statement is open to interpretation, or the context of the statement provides conflicting cues. The coding manual also contains samples of manifestos for reliability testing.
The coders involved in our study were trained using the manual. As already mentioned, we opted to retain the structure of quasi-sentence divisions imposed by the CMP. This means that our coders accepted how the manifesto text has been parsed into quasi-sentences and did not attempt to parse the text into a different combination of quasi-sentences. To establish a degree of inter-coder reliability for our coding scheme, all manifestos for Moldova and Romania were coded twice by two different coders.
Party-positioning on ethno-cultural issues: New estimates
In this section, we report the substantive results of our coding exercise and compare our estimates with those of the CMP. We also provide details of inter-coder reliability tests and discuss issues related to the validity of our estimates. All estimates generated by our project can be downloaded from the project website. 7 On the website, we also report standard errors of our estimates as recommended by Benoit et al. (2009).
Our findings provide strong support for our initial expectation that the CMP systematically undercounts the number of ethnicity-related statements and thus underestimates the importance of ethno-cultural issues in party competition. This expectation was confirmed by the examination of party manifestos in each of the four country cases. We first report the counts of ethnicity-related statements aggregated across all party manifestos on a country basis. For manifestos that we analysed, we identified almost twice as many ethnicity-related quasi-sentences as the CMP reports under its seven ethnicity-related categories. Graph 1 provides details on raw frequency counts. It shows under which CMP categories ethnicity-related statements were found and how the overall count of these statements was distributed across categories and countries.
As the graph indicates, the ethnicity-related statements we identified were originally coded under many CMP categories, including a large number of non-ethnic categories. The same CMP non-ethnic categories served as a host for ethnicity-related statements in party manifestos of more than one country. These findings fit into the CMP’s earlier conclusion that there are ‘highly populated’ categories among the variables in the CMP coding scheme (Volkens, 2001). These are the variables that are defined in such a general way that their analytical utility is compromised. Some of these categories became residual in the sense that there was a tendency for the coders to put a vaguely stated or cumbersome statement in one of such categories.
Some of the ethnicity-related statements we identified and presented in Graph 1 also provide a good illustration of the limitations of the CMP’s approach to the difficult problem of accounting for multiple membership of manifesto statements in several policy categories. For example, variables ‘301 – decentralization: positive’ and ‘302 – decentralization: negative’ have been used for coding both statements about decentralization as a system of delegating power to local authorities and also statements about creating territorial autonomies for territorially concentrated ethnic minorities. From the perspective of ethnic politics these are two different policy issues and party stance on these two issues might differ.
Our finding that CMP undercounts the number of ethnicity-related statements is not just a problem for specific parties. The fact that undercounting has been a systematic problem across the entire spectrum of political parties is illustrated in Graph 2. The graph uses a percentage scale to demonstrate the difference between the CMP’s and our results of the salience of ethnicity-related statements for all parties in our country samples.
The graph indicates that undercounting was a problem for the vast majority of party programmes covered by the CMP. For a number of manifestos, our estimates do not simply augment the CMP-determined salience of ethnic issues, but actually introduce these topics in the discussion of party-positioning – this is the case when the original CMP estimates are equal to zero.
Our estimates also significantly change the ordering of parties in terms of the salience of their positions on ethnic issues. In the case of Moldovan parties, for example, our estimates reverse the positions of four parties with the highest salience of ethnic issues. These parties ran on minority-friendly programmes that contained a large number of positive multicultural statements. Ours and the CMP’s estimates agree that ethnic issues in the Moldovan case were the most salient for these four parties, yet we disagree on their individual standing relative to each other. These disagreements have significant implications for the substantive discussion of party-positioning and the evolution of party competition over time.
Our results from applying the three-category classification scheme of the multicultural dimension and comparing our findings with those of the CMP are presented in Graph 3a. Since our data are the most comprehensive in the case of Moldova and Romania, we report here the results for these countries only.
The first two bars in Graph 3a compare the CMP’s and our results of classifying the CMP-counted ethnic statements. These are the statements that were coded under the CMP’s seven ethnic categories which we listed earlier in the article. As the graph indicates, integrationist statements have minimal presence in the CMP’s estimates and the vast majority of ethnic statements were coded by CMP under the ‘multiculturalism: positive’ category. Our results suggest that integrationist or, in the CMP’s terminology, ‘multiculturalism: negative’ statements were much more frequent than the CMP’s analysis leads us to believe. Neutral statements were found to comprise quite a substantial share in both country cases presented in the graph.
The bottom bars in Graph 3a give details on the distribution of statements in our counts of ethnic statements. These counts are based on a complete set of all ethnicity-related quasi-sentences identified by our coders. For both country cases, about a third of quasi-sentences that belong on the multicultural/integrationist dimension were coded as neutral. The share of integrationist statements was also substantial and many times higher than the CMP’s estimates. These results indicate that neither neutral nor integrationist categories are ‘empty’ categories and therefore should be taken seriously in any manifesto-based analysis of party competition on ethnic issues.
Graph 3b gives the results of our attempts to differentiate among types of positive multicultural claims. The graph is based on our counts of positive multicultural quasi-sentences. Identity preservation statements, which constitute the least politically sensitive category of multicultural claims, account for more than half of multicultural quasi-sentences in both countries. The remaining shares are composed of statements dealing with what could be considered higher level claims – those dealing with demands for group-based political representation, territorial autonomy and constituent nation status. Neither of the higher order categories, however, was found to be empty for either of the country cases. This encourages us to believe that the proposed typology could be usefully applied for understanding the nature of multicultural claims put forward in different national contexts.
Reliability and validity of our estimates
The critical final step of our analysis consists of testing the reliability and validity of our measurement scheme and party position estimates. To test for reliability, the following procedure was adopted. All Moldovan and Romanian party manifestos included in this study were coded by two different coders. A comparison of estimates produced by different coders for a large number of party manifestos constitutes the basis of our efforts to evaluate the degree of inter-coder reliability in using our approach (Krippendorff, 2004).
For calculating reliability scores, we used Krippendorff’s alpha for reliability, a formula which calculates the percentage of agreements between different coders, by also taking into account the relation between the observed and expected disagreements (2004: 221–3). Krippendorff established the value of 67 percent as the minimum acceptable agreement score for the reliability test (2004: 241). In our case, we obtained a 77 percent agreement score for Moldovan manifestos and 72 percent for Romanian. Overall, these results give us confidence that the procedures we employ for assessing the degree of ethnic issue politicization in party manifestos produce reliable estimates.
We also rely on Krippendorff (2004) in our discussion of validity tests. Krippendorff suggests that in order to empirically validate data obtained through content analysis one needs evidence that justifies the treatment of the text (content validation), evidence that justifies the inference that the content analysis is making (internal validation), and also evidence that justifies the results obtained (which, for the sake of simplicity, we label external validation). Below, we discuss how our method and generated scores perform on each of the three types of validation.
Content validation requires evidence on sampling and semantic validities. The former determines the degree to which the sample of texts used for the content analysis represents the entire population of texts from which the sample is drawn. The latter determines ‘the extent to which the categories of an analysis correspond to the meanings these texts have within the chosen context’ (Krippendorff, 2004; 318–19). The sampling validity of our estimates is supported by the case selection principles we applied, as discussed in our Introduction. The semantic validity of our results is supported by the already introduced evidence that confirms the propensity of our coding scheme to capture key aspects of ethnicity-related competition relevant for the region. We identified the multicultural/integrationist divide as a primary dimension of party competition in the ethno-cultural domain. The relevance of this dimension for Central and Eastern European party systems is supported by various studies of political competition in the four selected countries (Birch, 1995; Bugajski, 2002; Crowther, 1997; Popescu, 2003). The relevance of the second key dimension we identified – the titular ethnic group dimension – for studying politics in the region is also prominently acknowledged in the literature (Brubaker, 1996a).
For the internal validation we use evidence that the estimates we generated fit generalizations and patterns identified in the academic literature. One important generalization that is relevant to our work has been articulated in Roger Brubaker’s well-known research on nationalism in Central and Eastern Europe. Many post-communist countries facing old or new challenges of state-building choose, in Brubaker’s view (1996a, b), ‘nationalizing’ policies at the start of transition. Given Brubaker’s observations, which are generally accepted in the literature, one might expect this ‘nationalizing’ or integrationist message to be reflected in parties' integrationist stances during the 1990s. The estimates we generated are well in line with this generalization. In contrast, the CMP’s estimates are not, because according to CMP data there were no integrationist parties during the 1990s in the countries we examined.
Finally, the external validation requires evidence that the estimates we generated are in line with the data produced by alternative sources. If our content analysis method performs better in determining parties' positions on ethno-cultural issues than the CMP’s approach, alternative data sources should reflect this. We rely here on two types of external data to address the issue of validity: expert surveys and qualitative case studies.
In the case of the Moldovan party system, there is plenty of expert survey data for the time periods covered by our study. Surveys by both Benoit and Laver (2007) and Protsyk et al. (2008) contain relevant estimates. Our results agree with both these surveys on cases where the CMP and our estimates of party-positioning differ significantly; for example, in evaluating integrationist positioning in the 2001 Moldovan parliamentary elections. The CMP data suggest that there were no parties positioning themselves at the integrationist end of the continuum. Our estimates indicate that one of the electorally successful parties – the Christian Democratic Party of Moldova (PPCD) – had a clearly articulated integrationist stand. Similarly to the above mentioned surveys, the academic case studies of the Moldovan party system contain a similar interpretation of the PPCD position (Bugajski, 2002).
Benoit and Laver’s survey is also relevant for our examination of party competition in the Romanian case. These survey data indicate that one of the Romania parties – the Great Romania Party (PRM) – had a radically nationalist position around the time the survey was conducted. A similar interpretation is prominent in case study research (Popescu, 2003). Our estimates of party-positioning in the 2000 Romanian parliamentary elections confirm this finding. At the same time, the CMP data reveal no party with a strong integrationist position in Romania during that period.
Similarly, the CMP data fail to take account of the integrationist stances of political parties in earlier parliamentary elections in Moldova and Romania. Graph 4 provides an illustration of how the CMP and our estimates differ for the 1994 Moldovan and the 1992 Romanian parliamentary elections.
The CMP data suggest that there were no integrationist parties among the winners of the 1994 Moldovan elections and 1992 Romanian elections. Our estimates prove the contrary. The case studies of both party systems contain interpretations in line with our findings (Bugajski, 2002; Crowther, 1997). As Graph 4 also indicates, the CMP and our estimates differ for the multicultural part of the dimension as well. Questions about whose estimates of the multicultural positions are more precise could not readily be addressed with the sources we examine in this study.
In the case of Bulgaria, our research produced estimates of party positions only for the early 1990s. Kitschelt et al.’s (1999) party elite survey is relevant for validating our estimates for the Bulgarian party system. Among other things, the authors highlight their finding that the Bulgarian communist successor party – the Bulgarian Socialist Party (BSP) – adopted a much more nationalistic position than communist successor parties in other countries covered in the project. This is in line with what our examination of the Bulgarian party system revealed. While our estimates for the BSP indicate that the party is on the integrationist side of the continuum, the CMP’s score on integration for the BSP is zero.
In the case of Ukraine, our research produced estimates for parties competing in the 1994 parliamentary elections. Secondary sources identify several parties as strongly ‘nationalist’, which we usually interpret as an indication of a potential integrationist position (Birch, 1995; Bugajski, 2002). Our estimates for this country case do not agree very well with these secondary sources: our method produces a high integrationist score only for one of these parties – the Ukrainian Conservative Republican Party (UKRP). This disagreement between our estimates and the secondary literature is, to a large extent, due to the way we approach party positions on multicultural and titular group dimensions: we differentiate these positions, while the examined secondary literature conflates them. Rukh, a prominent pro-democracy movement in the early 1990s, for example, transformed itself by the time of the 1994 elections into a party that both favored the strengthening of a titular group identity and advocated minority-friendly policies. Finally, the CMP method registers an integrationist position for neither of the parties that took part in these particular elections.
Conclusion
We have put forward a proposal on how students of ethnic politics could analyse the politicization of ethnic issues in party manifestos. Our method for human-based coding seeks to address some of the limitations of the framework used by the CMP. Similarly to the CMP, we see the value of having, in a toolbox available for researchers, a content analysis method that relies on human coders. We argued that our approach helps to address some of the problems associated with the CMP’s method. The latter routinely underestimates the salience of ethnic issues in party manifestos and provides a very limited repertoire of means for differentiating among different types of ethnic claim.
We have tried to demonstrate that using our method allows for the generation of valid and reliable estimates of party-positioning on ethno-cultural issues. In particular, our coding scheme helps avoid the situation faced by CMP coders in which more than one coding category can be used to classify a statement. This is a source of many reliability problems for the CMP. Our approach also enables researchers to conduct a much more nuanced mapping of ethnic competition; it both counts and evaluates ethnicity-related statements. It retains such advantageous properties of the CMP’s basic framework as cross-party, cross-country and longitudinal comparability. It does not, however, allow for comparison of party positions across ethno-cultural and other policy domains. The CMP’s method provides for such comparison by excluding the possibility of individual statements belonging to multiple categories. Yet, as we have demonstrated, the CMP’s approach of forcing a statement to belong to only one coding category has considerable costs.
The coding procedures that we advocate require thorough examination of manifesto texts. These are resource-demanding procedures, but current alternatives for generating estimates of party positions on ethno-cultural issues do not provide a satisfactory solution. The intellectual payoffs from engaging in such an examination are substantial. As noted in the Introduction, the estimates generated by the method we propose can be used to explore a number of themes central to different strands of literature on politics in ethnically diverse societies. These themes include, but are not limited to, the dynamics of ethnic mobilization and the role that ethnic and mainstream parties play in the escalation/de-escalation of claim-making; the patterns of ethnic outbidding, polarization and cooptation; the forms of diversity accommodation, coalition-building and substantive representation. The proposed estimates provide a way of operationalizing concepts that are used in a different analytical capacity of dependent, independent or intervening variables in explanatory accounts articulated under these themes. To provide just one illustration, a systematic testing of the ethnic outbidding thesis – more intra-group elite competition leads to less moderation in group demands – requires accurate and reliable measures of the intensity of claims articulated by ethnic elites across different groups and over time.
As our findings indicate, ethno-cultural issues are more politically salient than the CMP estimates suggest. Many parties that operate in ethnically diverse societies are forced to take a position on ethno-cultural issues. Party manifesto analysis can help to structure inquiry on such position-taking and can improve our general understanding of how ethnic issues become politicized. At the moment, party manifesto analysis remains a severely underutilized tool by students of ethnic politics and we hope that the approach we propose can contribute to changing this situation.
Footnotes
Appendix I
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Acknowledgements
We are grateful to Andrea Volkens, Sonia Alonso, Monica Caluser, and Daniel Bochsler for their advice. Andrea Volkens also provided us with the Comparative Manifesto Project's original data. Elena Ceban, Evgeniya Bakalova, Olena Chepurna, Adrian Pandev, and Konstantin Sachariew contributed valuable research assistance at different stages of this project.