Quantitative text analysis and the study of EU lobbying and interest groups

Dimensionality of the EU policy space

A first challenge when applying quantitative content analysis to the study of EU lobbying is analysing a policy space that has more than one dimension. Current applications of quantitative text analysis to research EU lobbying used the complete text of a position document to estimate one policy dimension only (Klüver, 2009). ² Yet, the realities of EU policymaking in general, and those of the EC consultations in particular, speak of a policy space that has more than one dimension. An attentive reading of both the Commission’s consultation calls and of interest groups’ submissions reveals that the drafting of EC legislative proposals revolves around several, substantially different (although of course interrelated to a certain extent), policy issues that can be clustered into different dimensions of one proposal. Within the EC consultations, an issue refers to those policy aspects on which the EC asks for stakeholders’ input. In most policy contexts ‘issues are difficult to identify’ and ‘may be aggregated in different ways’ (Baumgartner and Leech, 1998: 39–41). However, within EU policymaking, issues are relatively easy and straightforward to identify, based on the usually very detailed official documentation accompanying policymaking events. Previous research identified between 14 and 24 issues per policy proposal: for example, König and Pöter (2001: 340) indicate that ‘[t]he proposal on Free Movement consists of 22 issues, Subcontracting of 14 issues, Pregnant Women of 18 issues and Working Time of 24 issues’. Similarly, we identified 10 issues discussed in the consultation on CO₂ emissions. Given this high number of issues, it is fairly unlikely that they can be collapsed into one dimension. In our empirical analysis we find that the 10 issues identified in the consultation on CO₂ emissions correspond to two policy dimensions.

Information provision in analysed texts

A second challenge follows from the assumption that the analysed texts are directly related to, and provide information about, the relevant policy dimension(s). If EU position documents provide information about more than one dimension, then treating these documents as texts referring to one dimension when applying quantitative content analysis is problematic and might affect the construct validity of position estimates (Slapin and Proksch, 2014: 136). A case in point is the following example: our hand-coding of the consultation on CO₂ emissions revealed that seven organizations (Associacio per la Divulgacio de les Tecnologies Sostenibles – ADTS, The Spanish Association for the promotion of Electric and non-contaminated vehicles – AVELE, the European Association for battery, hybrid and fuel cell electric vehicles – AVERE, the European Tyre and Rubber Manufacturers’ Association – ETRMA, the International Energy Agency – IEA, Micheline, Shecco) did not express any specific positions on the considered issues, but instead made general remarks about how environmental standards and emissions levels can be improved. However, the quantitative content analysis indicates that three of these organizations had the most pro-environmental stances (ADTS, AVELE, AVERE), while one of them (ETRMA) had a very close position to that identified for the EC post-consultation, indicative of lobbying success (Klüver, 2009: 543).

Another fundamental challenge in this respect stems from the fact that quantitative text analysis estimates positions based on the frequency of words in texts. This is a key aspect when analysing party manifestos and political speeches, by their very nature ideological texts in which the frequency of specific words has a substantive meaning. However, this aspect is problematic when analysing EU policy position documents because these texts employ a technical language that does not use the emphasis of certain words (by repetition) as a way of conveying a political or ideological message. Both EU scholars and practitioners agree on the technical nature of European policymaking, documentation and legislative acts, in which a significant amount of relevant information is transmitted with the help of few technical words. In some cases, positions are articulated entirely with the help of numbers that convey the substantive difference between positions expressed in documents. Quantitative content analysis does not capture these differences because it estimates positions based on word counts and not based on the observation of differences in the numerical values stated in the documents. A difference in numerical values is sometimes the key aspect allowing us to differentiate between groups’ positions, as illustrated below. Moreover, the removal of such numerical information (an approach usually used when applying quantitative content analysis) might result in over-simplified (and potentially imprecise) estimations of positions, based on partial/incomplete information.

In our empirical analysis, one relevant example is the issue of how to reach the target of 120 g/km for the reduction of CO₂ emissions. Different positions were expressed, among which we note the following: lowering the target to only 233 g/km; a 135 g/km reduction resulting from motor technological improvements, plus a further 15 g/km reduction coming from other technological improvements and fuel efficiency measures; a reduction of 130 g/km resulting from motor technological improvements, plus a further 10 g/km reduction from other technological improvements and fuel efficiency measures. The removal of numbers from the text prevents, for example, an adequate estimation of ACEA’s position on this issue, which was expressed in the following way: ‘ACEA supports reaching the Community target of 120 g/km by a reduction to 135 g/km via car/engine technology and a reduction of 15 g/km via other measures.’ This illustrates a relevant characteristic of EU documentation that is not adequately matched by current applications of quantitative text analysis techniques to the study of EU lobbying.

The text generating process

We contend that the requirement of quantitative content analysis that all analysed documents underwent a similar text generating process also raises challenges when considering its application to EU position documents. These texts are authored by organizations that differ considerably in terms of interests represented, organizational structure and communication purposes. Organizations participating in EC consultations are diverse: business, NGOs, local authorities, etc. They lobby as Euro-federations, national associations or individual organizations. Previous research indicates that EU interest groups employ position documents in different ways and for different purposes: some submit them to express their positions on issues the EC consults them on; others use these documents just to signal to decision makers their stakeholder status and to show their constituency that they are performing their interest representation mandate (Bunea, 2014: 1227). They have different organizational environments and resources. BusinessEurope has, most probably, a different approach to generating a position document to the Royal Society for the Protection of Birds (RSPB). Some organizations use highly technical language, while others do not. We observed this to be the case in the consultation on CO₂ emissions: organizations representing the car industry employed a more technical language and were more likely to use tables and figures to formulate their documents than environmental NGOs.

Current applications of Wordfish to EU lobbying research emphasize the importance of interest groups’ organizational form and distinguish explicitly between membership and non-membership organizations (Klüver, 2009: 540). The organizational form represents thus the criterion on which current Wordfish analyses excluded eight position documents submitted by eight organizations that did not comply with this (rather inadequate in our view) definition of an interest group in the context of EU lobbying. This distinction is however particularly relevant for and supports our argument regarding the diversity of the generating process of EU position documents: it directly implies that there are fundamental substantive differences between EU lobbying organizational actors authoring these texts. These differences translate into text generating processes that are not similar: for example, EU level associations face other constraints than individual firms when articulating their positions due to their more complex intra-organizational environments in which they have to reconcile the ‘logic of membership’ to that of ‘influence’ (Beyers, 2008). We also note at this point that several of the individual business actors that were excluded from the Wordfish analysis because they did not comply with the aforementioned definition (e.g. General Motors (GM), Micheline, Volkswagen United Kingdom), submitted documents that included to a relevant extent technical information conveyed with the help of tables and figures.

Therefore, it is reasonable to argue that the text generating processes vary across organizations according to their advocate type and organizational form. This also means that these documents are not comparable to each other, an aspect which is at odds with the fourth requirement of quantitative content analysis.

We note that it is equally problematic to apply the same estimation technique to interest groups’ position documents and to ‘press releases accompanying the [EC] communication and the adoption of the proposal’ as currently done in the literature (Klüver, 2009: 539). These are two different and non-comparable categories of texts generated by two types of actors (interest groups and decision makers), within very dissimilar intra-organizational environments and subject to differing ‘institutional constraints’ (Proksch and Slapin, 2012).

Comparability of analysed texts

A further reason why EU position documents are not comparable texts amenable to quantitative content analysis is their linguistic regime. Quantitative content analysis requires a unique language across texts. For example, applying Wordfish explicitly requires that all documents are written in the same language. In the EU, many organizations choose to make submissions in their native language and therefore the application of quantitative content analysis usually means excluding all submissions made in languages other than English. This can often lead to the exclusion of potentially relevant observations about individual organizations. For example, this implies that German organizations making submissions in German cannot be included in quantitative content analyses together with English submissions, despite the fact that these groups form one of the key players in the design of European policies. This suggests a biased approximation of the number of relevant data points included in the subsequent empirical analyses investigating determinants of lobbying behaviour and lobbying success within a certain community of stakeholders, and a limitation of the certainty with which causal inferences are made. This also raises concerns about the validity of position estimates extracted based on quantitative text analysis, given that these estimates are calculated upon (and therefore sensitive to) the total number of submissions analysed.

The number of analysed actors is particularly relevant when considering one of the main aspects recommending quantitative content analysis: its ability to facilitate large-n analyses. Yet, in practice, the method seems to systematically reduce the actual number of observations, because of its linguistic and text requirements. For example, applying Wordfish to a study of the consultation on CO₂ emissions implied loss of information from about 45% of the interest groups lobbying on this consultation, either because of Wordfish’s linguistic and text requirements or because they were authored by organizations that participated in the consultation as an individual firm, international organization or local authority (i.e. non-membership organization). We also note that the documents generated by such actors consisted on average of more technical content. We identified 45 organizations submitting individual or collective position documents, whereas the Wordfish based study examined only 25 following the exclusion of documents based on considerations related to language, organizational form and document text format (see Klüver, 2009: 540). In total, the Wordfish analysis excluded 10 organizations because their individual or joint submissions were not in English and eight organizations because of their organizational form. Two submissions made by two interest groups were excluded because of their text format: one submission (authored by the European Aluminum Association) was excluded because it was presented as a power point presentation in a .pdf file, while another submission was at that point in time (2009) protected by a password. ³ These particular two last exclusions provide in our view additional evidence about the diversity of texts submitted as position documents in the EU consultations. This heterogeneity is at odds with the assumption of comparable texts intrinsic to unsupervised text scaling algorithms.

Last but not least, the substantive differences among EU policy domains in terms of characteristics of policy issues, dimensionality of the policy space, technicality levels, specific terminology and type of organizations participating in policymaking raise further challenges relative to the assumption that interest groups’ position documents are comparable texts across policy areas, suitable for quantitative content analysis.

An alternative view: Qualitative content analysis

We ground our empirical assessment of quantitative content analysis applied to the study of EU lobbying in a research design that relies on the human coding of groups’ position documents submitted to the Commission when drafting a regulation on the reduction of CO₂ emissions from passenger cars (COD/2007/297). We selected this case because it was also used in the scholarship to validate Wordfish as a suitable method of quantitative content analysis to research EU lobbying (Klüver, 2009).

We used the position documents to identify policy issues and interest groups’ positions. To identify the relevant information, our unit of analysis was text paragraphs in the document addressing a specific issue and describing the group’s position. Employing text paragraphs as a unit of analysis was informed by two reasons. This approach follows the imperative to select a relevant unit of analysis based on the context of research (Roberts, 2000: 268). Next, to pay attention to the context of research focusing on EU policymaking and involving EU documents means taking into account their high technicality. Using text paragraphs as a unit of analysis instead of words, natural sentences or clauses, is therefore considered a more suitable approach. The informative value of text paragraphs when analysing EU official documentation is illustrated, for example, in the scholarship on EU compliance studies. These studies include text recitals from the EU laws as a measure of the information intensity, or complexity, of the European legislation (Thomson and Torenvlied, 2010: 151). Most commonly, in the analysed documents, one or several paragraphs were used by organizations to express their position on one issue. These paragraphs were usually part of titled sub-sections that announced that there is a relationship between the formulated text paragraphs and the issue to which the title of the section refers to. A group’s position on one issue was identified by taking into account all relevant paragraphs providing information with respect to that issue. In most cases, the position was succinctly stated in one or two key sentences contained in the considered paragraphs, and these informed our qualitative judgment and coding. We identified the position expressed by the interest group and assigned it a value. Organizations articulating the same position received an identical score on the position variable in our dataset. ⁴

We define a policy issue as a discrete policy problem on which the EC explicitly asks for policy input. We identified 10 issues for the consultation on CO₂ emissions:

Measures for reaching the target of 120 g/km for the reduction of CO₂ emissions;

Figure 1 presents the aggregate distribution of positions across issues and organizations. OPEN IN VIEWER

A policy position refers to a formally expressed support for a certain policy outcome. It denominates an observed position or expressed/revealed preference of an actor (Benoit and Laver, 2006: 11–34). For each issue, we identified all substantively different positions articulated by organizations in their submissions. We constructed an ordinal scale to estimate positions. Each position received a value indicating that it is substantially different from the others while expressing the rank of each position relative to each other in terms of the underlying policy dimension characterizing each issue. Our measure indicates that the positions are substantively different, and in which way they differ on the policy continuum described by each issue, but they do not indicate by how much. The policy distance between them cannot, therefore, be assessed. Figure 2 exemplifies our concept of policy position. OPEN IN VIEWER

Based on the text of the EC proposal drafted post-consultation, we identified the policy outcomes. In a similar manner to estimating positions, outcomes were coded based on text paragraphs from the Commission’s proposal. This allows us to estimate interest groups’ preference attainment during the EU policy formulation stage: we looked at the correspondence between a group’s position and the outcome (Dür, 2008). We first coded a dichotomous measure of preference attainment, indicating for each organization whether or not it achieved its position on each issue. In our analyses we aggregated this into a continuous measure of preference attainment, indicating for each organization the share of positions achieved out of the total number of positions expressed across issues. ⁶

Our content analysis approach presents several advantages when researching EU lobbying: it allows capturing salient characteristics of EU position documents, such as their technical content and diverse linguistic regime; it permits applying a similar coding scheme and analysis to two different types of documents that resulted from two different data generating procedures (interest groups’ documents and EC proposals); it facilitates differentiating between groups’ positions based on their substantive meaning; it results in disaggregate measures of both issues and positions which can than serve different levels of analysis, depending on research goals and designs; it increases the accuracy with which organizations are identified as actors in policy events by allowing the analysis of submissions made in different languages; it enables the coder to identify and discern between informative and non-informative documents with respect to the dimension of interest.

Thus, our approach performs well on those points where quantitative text analysis fails to meet the specificities of researching EU lobbying. However, as any qualitative content analysis, our strategy suffers from a set of limitations: it is time consuming and labour intensive, and its systematic application to large-n research designs requires patience, time, a serious deployment of research funds and a body of well-trained coders that possess a solid and thorough knowledge of the analysed policy events.

Validity of position estimates

As a first step to check the validity of position estimates, we plotted the Wordfish scores in relation to the hand-coded ones for each of the ten issues we identified, and present this in the online Appendix accompanying this article. The plot revealed some commonalities (i.e. a dichotomous distribution of positions on some issues indicated by both methods), as well as some significant differences. For example, across several issues, organizations that we identified to have expressed the same position are placed by Wordfish on different points of the policy continuum, sometimes even at the opposite ends. On three issues, Wordfish estimates place car producers’ organisations on opposite sides of the spectrum: the European Automobile Manufacturers’ Association (ACEA), the Japanese Automobile Manufacturers’ Association (JAMA) and the British Society of Motor Manufacturers and Traders (SMMT) are placed on the negative side of the spectrum (anti-environment) while RAI Association (the Dutch Automotive Association) is placed by Wordfish on the opposite side (pro-environment). Similarly, Wordfish indicates that the Association of German Car Manufacturers (VDA) took the most anti-environmental policy stance. This diverges significantly from what we find: while VDA did indeed ask for lower environmental protection standards, it was not the avant-garde of anti-environmentalists. VDA expressed only three positions, on a total of 10 issues that could be labelled ‘anti-environmental’, while GM and ACEA articulated respectively six and five positions on 10 issues asking for lower environmental standards. These two organizations adopted thus a stronger anti-environment stance, indicated by a higher number of positions demanding lower environmental control.

As a second step to check the validity of position estimates, we computed a measure of association between the Wordfish scores and our hand-coded positions for each of the 10 issues, reported in Table 2. On some issues, we notice a relatively high level of association between the two methods, which is explained by the fact that some issues generated indeed a dichotomous positioning of organizations on the policy spectrum, as attested by our qualitative coding (e.g. issues two and three). However, on other issues we notice lower association levels, which may be explained by the fact that, as detailed below, the issues were belonging to a second policy dimension that Wordfish scores do not capture appropriately (e.g. issues seven, eight and nine). This relates directly to the first methodological challenge we identified in section one, namely the necessity to capture appropriately the dimensionality of the policy space.

OPEN IN VIEWER

Table 1.

Spearman rho correlation coefficients of Wordfish and hand-coding estimates of policy positions.

Issue no	Spearman rho
1	0.47*
2	0.72**
3	0.72**
4	0.61*
7	0.65
8	0.50
9	0.09
10	0.76**

OPEN IN VIEWER

Table 2.

Explaining preference attainment.

	Model 1	Model 2	Model 3
Intercept	−0.09 (0.73)	0.60** (0.16)	0.47* (0.20)
Reference category: car producers’ business organizations
Non-car business	0.91* (0.37)	−0.35* (0.16)	−0.47 (0.31)
Environmental NGOs	0.65 (0.48)	−0.02 (0.11)	0.15 (0.17)
Others	0.04 (0.51)	−0.02 (0.13)	0.13 (0.20)
Opposition	0.67 (0.46)	−0.28 (0.24)	−0.19 (0.26)
Included in Wordfish?			0.13 (0.15)
Non-car*included			0.14 (0.30)
Environmental*included			−0.31 (0.20)
Other*included			−0.23 (0.26)
R 2	0.30	0.20	0.30
Adj. R2	0.16	0.10	0.11
No. of obs.	25	38	38

Note: OLS regression models. Reference category: car industry organizations. Standard errors in parentheses. The dependent variable in Model 1 is the absolute difference between the position estimate of the organization and that of the Commission post-consultation. The dependent variables in Models 2 and 3 are the share of the issues on which the position of the organization coincided with the position of the Commission post-consultation from the total number of issues on which this organization adopted a position. *p < 0.05, **p < 0.01.

To uncover underlying patterns of groups’ positioning across issues and corresponding policy dimensions, we analysed hand-coded positions of interest groups using Specific Multiple Correspondence Analysis (SMCA). The Multiple Correspondence Analysis is a well-established method for analysing nominal data, such as the coding of groups’ positions used in this research. The specific MCA is a variant of MCA restricting the analysis to selected categories only (Le Roux and Rouanet, 2010: 61; Greenacre (2007: 161) refers to this method as ‘subset MCA’).

Before presenting our results, we note that, in comparison to other consultations organized by the European Commission, the consultation on the CO₂ emissions from passenger cars is more likely to be characterized by a unidimensional policy space. The number of organizations that participated in this consultation was relatively low (less than 50) compared to many other consultations that involve hundreds of stakeholders. Moreover, the range of issues covered by this legislative act was also relatively narrow by comparison to other consultations: for example, the consultation on the waste electric and electronic equipment (2008) revolved around 25 issues. Also, the organizations that participated in this consultation represented mostly business interests in one particular economic sector (car manufacturing) and environmental NGOs. The diversity of interests and expressed positions was thus potentially lower than that of other consultations. Thus, if, despite these contextual features, the policy space in this consultation has more than one dimension, it is likely to be multidimensional in many other consultations as well.

We restricted the analysis to non-missing positions only. In other words, the cases in which organizations did not express a position on a specific issue do not affect the results of the analysis. Twenty-four different positions on 10 issues were adopted by the actors participating in the consultation. ⁷ We excluded five positions expressed by a single actor only; such categories with very low frequencies can contribute substantially to the determination of the axes derived from the analysis, which may impede the interpretation of these axes (Le Roux and Rouanet, 2010: 61–62). The analysis included 19 positions on 10 issues. We implemented our analysis with the help of the soc.ca package in R (Larsen, 2014).

Figures 3 and 4 illustrate the results. The modified rates (measures of the total variance related to dimensions identified by MCA) of the first and second dimensions are 79.8% and 11.6%, respectively. The modified rate of the third dimension by contrast is only 5.9%. The first two dimensions therefore describe the data reasonably well. Thus we believe the two-dimensional solution suits the data best. ⁸ Figure 3 places the categories on the two dimensions. The plot on the left shows the placement of the 19 categories on Dimension 1 (x axis) and the extent to which these categories contribute to Dimension 1 (y axis). The plot on the right presents the same information for Dimension 2. OPEN IN VIEWER

OPEN IN VIEWER

Figure 4.

Interest groups’ positions on the two-dimensional policy space in the consultation on CO₂ emissions. Note: Organizations that were included in the Wordfish analysis: in dark black. Those excluded: in grey. For three organizations included in the Wordfish analysis hand-coded position estimates were not available.

The first plot suggests that the first dimension is related to the stringency of the regulatory regime aimed at reducing emissions with the help of ‘fuel efficiency measures’. On the left we identify the category that contributes the most to this dimension: it refers to an earlier and more ambitious deadline for reaching the reduction target (‘time.2012’), together with other categories asking for a stricter regulatory regime (e.g. ‘penalties.yes’). On the right are the positions that refer to a more relaxed regulatory regime. These positions proposed less ambitious technical measures aimed at motor improvements (‘target.2’), asked for an extended deadline to reach the target (‘time.2015’), advocated increased flexibilities for car producers (‘flex.increased’) and supported weight as a parameter for deciding which vehicles were covered by the regulation (‘param.weight’). This dimension corresponds to what the EC identified as supply oriented measures (European Commission, 2007: 8).

The second plot illustrates the second dimension. This relates to setting up an additional regulatory framework aimed to complement the aforementioned one, and consisting of a set of measures (fiscal and marketing oriented) to motivate consumers to buy fuel-efficient cars. On the left of its spectrum we notice such positions as the agreement with the establishment of an EU-wide car marketing code of conduct (‘marketing.yes’), no flexibilities allowed to car manufacturers (‘flexibility.none’) and more ambitious fuel efficiency measures (‘target.4’). The positions on the right side are characterized by the opposition to, or less support for, this additional regulatory framework: disagreement with the marketing code (‘marketing.no’), allowing moderate flexibility for reaching the target (‘flexibility.modest’) and the position that the emission target should be reached through basic fuel efficiency and motor improvements measures (‘target 3’). ⁹

Figure 4 illustrates the placement of organizations on these two dimensions and supports the notion of a two-dimensional space. On the first (x-axis), related to fuel efficiency regulatory measures, the environmental NGOs and the car industry are placed at opposite ends of the policy spectrum. This suggests their divergent views on how ambitious this regulatory framework should be. The estimates of positions on this dimension are quite closely related to the Wordfish estimates (r = −0.59). However, on the second (y-axis) the positioning of groups based on their advocate type does not follow a clear-cut pattern and one cannot distinguish between lobbying sides based on their advocate type. On some issues linked to this dimension, environmental NGOs and industry organizations expressed identical positions. The coefficient of the correlation between this dimension and Wordfish estimates is only 0.12.

Figure 4 also differentiates between 21 organizations that were included in the Wordfish analysis and 17 that were not included. This figure relates to the fourth point in our theoretical section where we discuss the issue of comparability of texts and the importance of having texts written in the same language, following the same text generating process by similar actors. The graph demonstrates that the exclusion of organizations required by Wordfish is not random with regard to their positions and interest type. While the mean positions of the ‘included’ and ‘excluded’ groups are roughly similar on both dimensions, 10 out of 17 ‘excluded’ groups were environment NGOs, leaving only 6 NGOs among the ‘included’ groups. Out of 10 ‘excluded’ organizations that submitted documents in French or German, nine were NGOs. Importantly, the policy positions of these NGOs excluded for linguistic reasons were substantially less extreme (and closer to the positions of the Commission) on the first dimension than the positions of the NGOs that submitted their documents in English (the mean values were −0.36 and −0.59, respectively), as clearly indicated in Figure 4. As explained below, these differences lead to substantively different results with regard to the hypothesis that environmental NGOs had lower preference attainment scores.

This illustrates why the estimation of a unidimensional policy space might be inadequate when researching EU lobbying and policymaking. While, similar to our analysis, Wordfish captures positions expressed in relation to the regulatory regime consisting of fuel efficiency measures (dimension one), it does not capture the second dimension referring to fiscal and marketing measures. The exclusion of some groups because of linguistic requirements or level of text technicality is not random, and can potentially affect the results of the studies aimed at understanding EU lobbying.

Validity of preference attainment estimates

Do these differences in the conceptualization of the policy space and the measurement of positions between quantitative and qualitative content analysis translate into different findings with respect to levels of preference attainment? We answer this by constructing a measure of preference attainment based on each of the two methods and we assess their correspondence (agreement) graphically (Figure 5) and with the help of regression analysis (Table 2). OPEN IN VIEWER

Preference attainment is most commonly conceptualized in the literature in the following manner: ‘[a]t its most basic, […] the distance between an outcome and the ideal point of an actor reflects the influence of this actor’ (Dür, 2008: 566). The Wordfish-based measure, which approximates this understanding of the concept closely, and also scores high on simplicity, is the absolute difference between the EC position and that of the interest group. Higher scores on this measure indicate that the position of the group was further away from that of the Commission (i.e. the estimated position score of the group was much lower or much higher than the estimated position score of the Commission), and therefore indicate lower preference attainment. ¹⁰

Based on our hand-coding we construct a measure of preference attainment that captures for each organisation the share of positions achieved out of the total number of positions expressed across issues: Share of preferences achieved. This is a straightforward measure estimated at organization level, which corresponds directly to the basic logic of the preference attainment approach – getting one’s position translated into outcomes.

We first examine the relationship between the Wordfish and our measure of preference attainment. We expect a negative correlation between the two, since higher scores on the Wordfish estimate indicate lower preference attainment. The regression line in Figure 5 shows some level of association between our estimates and the Wordfish measure but the strength of the relationship is modest (r = −0.3). We observe thus some level of association between the two measures. This is probably attributable to the fact that the two approaches share the policy dimension referring to fuel efficiency issues (dimension one above). Yet, the association is modest in strength because our hand-coding captures information about actors’ positions on what we identified as a second relevant policy dimension. This test challenges thus the validity of the Wordfish estimation of preference attainment.

To explore this issue in more depth, we take the analysis one step further and use both coding schemes on positions to examine a substantively important hypothesis suggesting that business organizations more successfully achieve their policy preferences. The dependent variables are the two measures of preference attainment we computed based on Wordfish and on our hand-coding. We use OLS models to explore interest groups’ levels of preference attainment. The main independent variable is a four-category variable, which differentiates between car producers, non-car industry business, environmental NGOs and other groups (local government, etc.). We expect that, all else being equal, organizations representing the car industry will have lower scores on the Wordfish measure (indicating a higher preference attainment) and higher scores on the measure derived based on hand-coding, relative to other organizations.

We control for the amount of opposition the group faced from other organizations when trying to achieve its preferences. In Model 1, this opposition variable is constructed as the average distance between the estimated position of the interest group and the positions of all groups for which estimates where available. In Models 2 and 3 (that employ a preference attainment measure based on our hand-coding), the opposition control variable is computed in two steps. First, for each issue in relation to which the organization expressed a position, the share of organizations that expressed substantially different positions is computed. Second, the mean of these shares is computed for each organization.

The results of our statistical analysis presented in Table 2 demonstrate similar effects of some variables, but important differences with regard to others. All three models provide some evidence that organizations representing the non-car business industry are less likely to achieve their positions compared to organizations representing car producers. Moreover, the coefficients of the opposition variables, although not statistically significant, have signs that are in line with the theoretical expectations (i.e. positive in Model 1 and negative in Models 2 and 3). Models differ mainly with respect to the difference between car manufacturers and environmental NGOs. Model 1 shows substantially lower levels of preference attainment for environmental groups (by 0.65, or 86% of the standard deviation of the preference attainment variable), even if the effect is not statistically significant. However, the effect of ‘environmental NGOs’ in Model 2 is very low and the standard error is more than five times higher than the value of the regression coefficient. This indicates that any important differences between the two advocate types are very unlikely.

While differences in preference attainment scores may be related to the two-dimensional space, which is not captured in the quantitative content analysis sufficiently well, it may also be a result of omitting some groups from the analysis. Model 3 explores this possibility. The dependent variable is our measure of preference attainment. The main independent variable is a dichotomous variable indicating whether the Wordfish estimate of the position expressed by the interest group is available. This dichotomous variable is interacted with the advocate type, in order to investigate whether preference attainment by different advocate types was over- or under-estimated.

The results provide relatively strong evidence that environmental NGOs included in the Wordfish sample indeed had a lower score of preference attainment than those excluded because of language (documents in French or German). The interaction effect between the category indicating ‘environmental NGO’ and the variable indicating the inclusion of the organization in the Wordfish analysis is negative, close to the 0.1 level of significance (t-test statistic is −1.55). Given the small sample size, this suggests quite strong evidence that French and German NGOs were more successful than their counterparts in other countries in translating their policy preferences into the policy proposal. The Wordfish analysis misses this important point by excluding these groups, which explains at least partially why Model 2 finds no difference between the car industry and environmental NGOs in terms of preference attainment.

We also attempted to investigate the extent to which different reasons for excluding groups from the Wordfish analysis (non-English documents vs. different organizational form) account for the lack of support to the hypothesis that environmental NGOs were less successful in attaining their preferences. However, the coefficients of the interaction effects between the dichotomous variables indicating groups excluded for each of these two reasons and the categorical variable indicating the interest type cannot be estimated due to the small sample size. Nevertheless, the evidence that environmental NGOs submitting documents in French or German were closer to the positions of the Commission (see Figure 4 and ‘Validity of position estimates’ section), suggests that omissions due to language explain to a larger extent why we find no support for the hypothesis that these organizations had lower preference attainment scores.