Relating news analysis and public opinion: Applying a communications method as a ‘tool’ to aid interpretation of survey results

Introduction

Large cross-national studies based on surveys of public opinion have often faced a problem: how to interpret findings to attitude questions and to some extent behavioural questions across different countries, when information is lacking about whether specific influential political events may have occurred that influenced public responses in some countries, but not in others, at the time of fieldwork. In the attempts to counteract this potential problem, there has been a concerted effort within the survey research community to record exceptional political events in different countries as a reference point for aiding the interpretation of survey data. Although attempts have been made to come up with a method of event recording that provides suitable reference data, none has proved sufficiently robust or comprehensive to be adopted widely. The initiative presented in this article documents a methodological approach that the authors, who have a communications background, developed through work and interaction with survey researchers, within the framework of the European Social Survey (ESS). ² We started from the premise that insights from news analysis in the communications tradition presented the best hope of producing a user-friendly tool for a systematic cross-national recording of ‘political events’ that would facilitate a more robust interpretation by survey researchers. In this article, we document the approach that we applied and provide a few examples to show how this works and improves on previous attempts.

The next section outlines our general approach for measuring the ‘political climate’ from news sources. This is followed by a detailed discussion of the claims-making approach and method (Koopmans and Statham, 1999) which was the key inspiration for our efforts. We then document how the claims-making approach was adapted for our purposes and linked directly to the ESS survey, before outlining a couple of examples, demonstrating how the retrieved data might be used as a resource by researchers. Finally, we discuss the implications of using media data in conjunction with survey data on public opinion and suggest further ways forward.

General approach – Measuring the ‘political climate’ from news

Before starting this project for developing a methodology, we were confronted by some earlier attempts by political scientists for coding ‘events’ that could be used in conjunction with surveys for identifying exceptional periods. Such attempts, including one conducted with the ESS, were not informed by communications approaches and methods. For example, the earlier ESS event recording ³ lacked a systematic and standardized definition for ‘events’ and set of codes. Researchers in the different countries simply wrote down what they considered to be the important events of the day. In some countries this was done in great detail, in others barely at all. Different criteria of selection and categorization were also applied in each country. This eclectic and random approach for defining, selecting, retrieving and reporting ‘events’ across the different countries placed serious limits on the potential of the data for identifying variations in ‘political climate’, either across rounds of the ESS survey, or between countries. Generally, we felt that the sophisticated level of questionnaire development in ESS that facilitated cross-national, cross-issue-field and diachronic analyses had not been matched by a method for recording ‘events’.

In addition, we felt that there had been an insufficient attempt to link ‘coded events’ to the specific substantive topics investigated by the ESS questionnaire. Not all of the ‘events’ identified by the earlier ESS event coding – e.g. sport or entertainment events – were relevant topics covered by the survey. Nor was the focus they were given when coded always relevant to the survey, e.g. an ‘event’ triggered by the Iraq War could potentially relate to ‘trust in institutions’ or ‘prejudice towards outgroups’, or, alternatively, none of the issues covered by the survey. So the coding of ‘event = Iraq’ told us nothing about the actual salience of issues and values that were measured by the public opinion survey. From this, we considered that it was necessary, first, to delimit the relevant issue-fields for coded ‘events’ to topics examined in the ESS survey, and, second, not to code ‘events’ in the colloquial sense, but the substance of issue-fields through which events are interpreted and become meaningful.

We aimed to develop a systematic method for retrieving more reliable information on the ‘political climate’ for use with the ESS by drawing from communications approaches and methods. In particular, we proposed to use the function of the media’s news selection processes as the basis for retrieving information on the ‘political climate’ with regard to specific issues and events. The agenda-setting perspective was seen as a useful starting point for our proposed method. The ‘agenda-setting’ or ‘agenda-building’ function of the mass media in relation to public opinion has been a central strand of communications research (e.g. Everett and Dearing, 1988; Hilgartner and Bosk, 1988; McCombs and Shaw, 1972). Media agenda-setting focuses on the ‘supply side’ of political information to the public. The selection of political events by the media and their production, representation and mediation in the form of news is seen to focus and shape public opinion on certain issues, at the expense of other possible alternatives, but not to directly determine the intensity of individual attitudes and opinions.

From the range of political issues and events that occur in the world every day or week, only a small selection becomes salient to the public. It is these salient events that have the potential for focusing and shaping the attention of members of the public, a sample of whom respond to survey questions. Via their selection and reporting of the news, the mass media are the primary conduit through which these salient political events are conveyed to the public. The media’s news coverage is a selection that focuses attention on certain political actors, events and issues, representing a record of the ones potentially most relevant to public life in a given place and time. Thus media reports are a good source for retrieving and categorizing them.

We decided to take ‘news’ as the important source of empirical data for identifying which specific events and issues were salient at a given time, and within a given country. When aggregated, this combination of salient events and issues would provide an empirically based measure for the ‘political climate’ of a country, at a specific time and place. By using a common schedule for coding the issues mobilized by actors in relation to events within national settings, it becomes possible to compare a ‘political climate’, cross-nationally, as well as changes within a single country over time.

In the methodological project documented here, we wanted to analyse news sources to measure the ‘political climate’ of countries during the phase of survey work. The first aim was to produce a standardized mechanism that could act as a ‘control’ for interpreting survey findings. For example, if cross-national comparison of survey findings produces surprising or unexpected results for a country on a specific topic, then the researcher may refer to the contextual data on ‘political climate’ to see whether that specific issue was especially salient, or prominent in an unusual and/or atypical way, at the time of the survey fieldwork. The methodological project was not conceived as an exercise in advancing the recording of political events from news in general, it was explicitly designed as a service provider for researchers using the ESS data. The unique feature of the project was that it related news contents and survey data together. More precisely, the logic of survey investigation provided the basis of the logic for coding the ‘political climate’. Our news analysis did not seek to record the ‘political climate’ generally, but only those key issues relevant to topics investigated in the survey. This might seem an obvious point, but, as we elaborate further later, it has vital implications for selecting and defining the issue-fields and events to be coded.

We selected newspapers as the important media source for retrieving information on significant national and international political events. Compared to other media, such as television or radio, newspapers produce a more detailed and more (cross-nationally) standardized format, and contain more of the type of elaborated political information (i.e. on how issues are framed) that we aimed to retrieve. There were also practical considerations. Newspaper sources allowed us to cover earlier survey periods, and enabled the actual recording and coding in different countries to take place at different paces. In addition, many newspapers from European countries were already easily and cheaply accessible online from Lexis-Nexis and where they were not we were still able to gain access to sources on microfiche.

Our pilot study for developing the method encompassed five countries: the UK, Germany, Ireland, Poland and Spain. By selecting a range of newspapers for each country, we aimed to retrieve political information that had appeared in different types of newsprint media (left/right; elite/mass audience). Trained coders then used a standardized coding scheme to retrieve information on the selected political issue-fields derived from the survey questions. This information was then entered into a database (using SPSS). Having done this systematically across countries, it was possible to interrogate the database in relation to specific time periods and/or issue-fields, in order to identify the characteristics of a national ‘political climate’ over an issue. In addition, it was possible to retrieve information on the salience of particular types of issue-fields relative to others.

So far, we have discussed the choices and aims of our approach, but provided little detail on the method for news analysis which provided the basis for our methodology. In the next section, we discuss the claims-making approach (Koopmans and Statham, 1999) in detail as this provides the conceptual and analytic foundations for our work. The subsequent section then clarifies some modifications to the claims-making approach that were applied to meet our specific goals of linking it to the survey. We then give a few examples that demonstrate how the method might be used by researchers in conjunction with the survey.

‘Claims-making’ as a method for retrieving political information from news

From a methodological perspective, news is a rich source for retrieving data on what Bennett and Entman (2001) call ‘mediated politics’: it provides information on which actors are able to successfully mobilize their political concerns publicly, what positions they take up over issues, the ideological contents of their arguments, who they address, support, and oppose, and whether this expands the debate, spatially, by communicating across national contexts and political levels. It is these claims-making political acts, or ‘claims’, which are visible in public spheres that we aimed to capture from newspaper sources and use as an indicator for the visible political agenda or ‘political climate’ that appears in the news reported in a country.

Claims-making analysis (Koopmans and Statham, 1999, 2010; Koopmans et al., 2005; Statham and Trenz, 2013) is an established method for examining the dimension of politics that is publicly visible. ⁴ The method analyses the contents of mass media news discourse. It developed initially as an extension to protest event analysis in social movement research (e.g. Rucht et al., 1999) by adding insights from public discourse (e.g. Donati, 1992), linguistic grammar (Franzosi, 2004), and framing approaches (e.g. Benford and Snow, 2000; Gamson and Modigliani, 1989), as well as related approaches from communications research (e.g. Entman, 2004; Neumann et al., 1992). The method is specifically designed to study political contestation that is produced by actors’ contributions to public debates carried by the media. Examples of this include reporting of political actors’ interviews, press conferences, or their publication of reports, and by other visible acts of political mobilization, such as protest demonstrations. In other words, the method retrieves ‘speech acts’ and ‘mobilization acts’ from news.

Many attempts by collective actors to mobilize political claims fail to reach the public domain. The limited carrying capacity of the media means that it has to select which events, which claim-makers, and which opinions, are newsworthy. For the type of questions that we address, however, it is the publicly visible claims that count, since this is the political information supplied by the media that is potentially accessible to the reading public in a specific country. Also, claims need to appear on the public stage to be able to resonate with the perceptions of other actors, and be able to input into processes of public opinion formation.

In many media content analyses conducted in the communications research tradition (see e.g. Krippendorff, 1980) the assumption is that the words and phrases mentioned most often are those reflecting important concerns. As a result these studies often analyse contents quantitatively starting with word frequencies, space measurements (column inches in the case of newspapers, time counts for radio and television) and keyword frequencies. An important difference between these approaches and the claims-making method is that the unit of analysis is not the ‘article’ but the ‘claim’. Content analysis methods using article-level variables can tell us with what frequency certain actors and issues are mentioned, and to what degree they co-occur in news stories. However, they tell us relatively little about the relationships between actors, or the positions that they take up with regard to which issues in the public debate, or their intensity and focus of their contestation. It is precisely this information, about who addresses whom, over which issues, and with what political demands, that we need to know to answer questions about the political information that is visible in a national public sphere. The claims-making approach focuses on the role of political actors and their claims in shaping the public debate over politics, which is seen as a field of contestation, and so the ‘claim’ not the article is the unit of analysis.

Another important distinction between claims-making analysis and much communication research comes from the type of research question addressed by content analysis. Even content analysis from the communication tradition that extends beyond word counts and quantification of space and time to analyse qualitative data, such as the sophisticated news framing approach (e.g. Entman, 2004; Reese et al., 2001), still starts out from the perspective of how social reality is constructed by journalists through news. By contrast, the claims-making approach takes news discourse as a source for analysing political contestation between a broad range of different types of political actors. This can sometimes include the media and journalists as a ‘political actor’ when they make their own ‘claims’ in the news, for example, through their claims in editorials (see e.g. Pfetsch et al., 2010).

Drawing from the insights of the classic research by Karl W. Deutsch (1953) that communication can structure the boundaries and relationships of a political community, claims-making acts can be seen as a set of communicative networks and relationships through which political actors interact, for example, within a polity, or across institutional levels of multi-level governance, or across national borders. To capture analytically aspects of the communicative networks generated by political actors’ mobilization of claims, the claims-making method extends beyond the traditional approaches in social movements and media research in several ways.

First, by extending the scope of retrieval to include a full range of actors, the method moves substantively away from the narrow focus in protest event analysis from the social movement research tradition on protests and protesting actors, which at best represent the ‘tip of the iceberg’ of public contestation. Second, by extending the type of communicative act that is included beyond ‘events’, claims-making analysis can capture much more, qualitatively and quantitatively, of the discursive contents of political contention. Third, claims-making analysis moves beyond the media-centrism of article-level content analyses, by retrieving information on the actor relationships that are reproduced and mediated in news. Fourth, drawing inspiration from Franzosi’s (2004) usage of ‘linguistic grammar’ in public discourse, claims-making analysis reconstructs contentious public debates by examining how actors establish communicative relationships through their public claims-making acts that target other actors, across political arenas, and spatial boundaries.

Overall, the claims-making perspective sees news as a record of public events and retrieves information on contention that is constructed by political actors in public. A political claims-making act, a ‘claim’, is a purposeful communicative action in the public sphere. It has been operationalized though the following definition: ‘Claim-making acts consist of public speech acts that articulate political demands, calls to action, proposals, or criticisms, which, actually or potentially, affect the interests or integrity of the claimants or other collective actors in a specific issue field’ (Koopmans and Statham, 2010: 55).

To give an idea of the type of information coded, a claims-making act can be broken down into six basic elements, and the method requires coding a number of detailed variables for each of these:

In a simple form, for a claims-making act: at a time and place (1) an event occurs, where an actor (2), mobilizes a speech act (3), which addresses another actor (4) calling for a response, that raises a claim about an issue (5), on the basis of a justifying argumentation or frame (6). This can be demonstrated by taking an example from a study of political contestation over the European Constitutional Treaty (Statham and Trenz, 2013: 173, italics in original) that was rejected in a referendum in France in 2005.

Importantly, in this method claims are included in the data sample regardless of who makes them and where they are made. Actors appear, not because they have been pre-selected, but only to the extent and in the way (e.g. in favour of, or critical of, a specific European integration issue) that they successfully make interventions in the mediated political discourse. This means that resultant data sets include claims by a wide range of state and institutional actors, economic and social interest groups, journalists and news organizations, as well as representatives of civil society and social movements. Claims can be made by organizations and their spokespersons, as well as by diffuse collectives, for example ‘a group of farmers’. The claim-makers may be from European, other supranational, as well as national, regional, and local levels, and they can be from the country where the newspaper is published, or from other countries. Thus, the data gathering strategy is neutral with regard to the geographical and political scope of claims, and actor types who make them, precisely because these are the aspects that are the objects for study.

A benefit of this analytic descriptive methodology is that it produces highly flexible data sets that allow for a combination of cross-national and cross-actor analyses, at different levels of aggregation. The method also allows for the study of qualitative detail of an actor’s claims; for example, how an actor frames justifications in making a claim, within the context of knowing the same actor’s relative share and position within the public debate.

Commonly aired objections to using media sources for data on political action refer to the role of the media in processing this information: the media’s selection bias (selection of which events and issues to report) and description bias (selection of the relevant information about events and issues to report). Against this though, the established tradition for taking newspapers as a data source for political contestation has produced a large number of studies assessing the impact of selection and description bias on the validity of newspaper data. Overall, such studies conclude that newspaper data do not deviate from accepted standards of reliability and validity, as long as one is interested in trends and differences rather than absolute numbers (Earl et al., 2004; Rucht et al., 1999). Possible limitations arising from the selection and description biases of specific sources of data can be reduced by drawing from more than one source. However, perhaps the most compelling argument in favour of using media sources is that the conceptual approach applied relies on the media’s news selection to give meaning to the data. Events and issues that go unreported or do not stimulate political claims are considered to be largely irrelevant: first, because the issue is not considered contentious enough to stimulate a mobilization or response by a political actor that journalists consider significant enough to cross the threshold of being newsworthy; and second, because events, issues and claims can have little impact in shaping the responses of other political actors or public opinion if they are not publicly visible and accessible to the public.

Adapting the claims-making method for use in conjunction with survey data

The claims-making method that we have outlined codes a large number of variables related to claims. Our requirements for this project were more parsimonious since our primary objective was to support survey analysis rather than reconstruct an image of the contents of a national public sphere. For that reason we reduced the amount of information coded to basic variables on: (1) the actor making the claim; (2) the issue-field; (3) the position of the claim in its evaluation of the issue (+1 in favour; −1 against; 0 neutral); and finally, the actual wording of the claim, in direct or indirect speech. In addition, basic details on the data of the claim, the newspaper it was reported in, and the headline of the relevant article were coded. Taken together, this data can be reconstructed to give basic information relevant to the ‘political climate’ on, for example, which actors make claims over an issue-field, the salience of different issue-fields relative to one another or over time, and what the substantive contents of the claims are. We demonstrate this in the next section.

The main important step that we made to link our news coding to the survey was to select and define our coded issue-field directly from the logic of questions in the survey. This requires further elaboration.

Although expansive in scope and depth, the ESS questionnaire does not attempt to reproduce information on the whole reality of all topics and issues. The logic of the survey has been shaped by key investigative decisions to cover specific aspects of social life and not others. This process of selection by the survey is a useful starting point for defining the political issue-fields that are relevant for coding from news. Our proposed strategy was to use the questionnaire as the basis for defining the universe of issue-fields to code from news. We coded only those political issue-fields that could have been potentially relevant and meaningful to survey respondents. Thus we were not interested in the whole universe of issues constructed around events that happen in the world, only those that are meaningful because they relate to topics pursued by survey questions. In short, the idea was that the logic of the survey questionnaire should define the basis for the relevant and meaningful coded issue-fields. Specific issue-fields were selected and defined in the code book so that they directly replicated the fields of inquiry of the survey questions.

It is worth recalling the aims of the ESS with regard to the information it seeks to retrieve on ‘public opinion’. Chapter 1 of the ESS Core Survey Questionnaire Development (available on the ESS website) defines the overall field of inquiry:

The central aim of the ESS is to develop and conduct a systematic study of changing values, attitudes, attributes and behaviour patterns within European polities … (T)he ESS aims to measure and explain how people’s social values, cultural norms and behaviour patterns are distributed, the way they differ within and between nations, and the direction and speed at which they are changing. (p. 3)

The core module aims to monitor ‘socio-economic, socio-political, socio-psychological and socio-demographic variables’ (p. 3).

From this broadly defined territory, the variables that were most relevant for our news analysis project were the ‘socio-economic’ and ‘socio-political’. By contrast the deeper individual-level ‘socio-psychological’ variables cannot be convincingly traced or determined from news contents, but are also presumably more fixed and less open to influence from short-term fluctuations in ‘political climate’. Also, the questionnaire’s ‘socio-demographic’ variables are about the structural location of the individual respondent and not connected to short-term shifts in ‘political climate’. Thus, our broad field to code was the ‘socio-economic’ and ‘socio-political’ dimensions of issues that actors constructed over political events that can be considered to constitute a country’s ‘political climate’ at a specific time.

Table 1 shows the specific issue-fields that we coded and gives the specific questions from the ESS core survey to which the issue-fields relate. This means that researchers examining specific questions can turn directly to the claims-making data and run quick queries on the relevant issue-field of their inquiry. To show how this works, so that a researcher can combine the information from the survey analysis to the claims-making data, we now provide some illustrative examples.

OPEN IN VIEWER

Table 1.

Selected issue-fields and their link to specific survey questions.

Issue-fields (summary codes)	Relevant survey questions
1 Reliance on media	A1–7
2 Community relations (non ethnic)	A8–10
3 Political engagement	B1–3 and B11
4 Trust in political institutions*	B4–10
5 Collective action/political mobilization by citizens	B13–19
6 Political party identity of actor	B20–23
7 Personal wellbeing and life satisfaction	B24
8 Perception of national performance	B25–29
9 State intervention (and its limits)*	B30–32
10 Environmental problems	B33
11 European integration (Enlargement/deepening)	B34
12 Immigration and ethnic/racial difference	B35–36
13 Immigration: economic impact	B37–38
14 Cultural diversity and its impact	B39–40
15 Crime/threat of violence/risk and perceptions of safety	C5–10
16 Terrorist act/threat	C11–12
17 Limits of liberal state’s responses (curtailing civil liberties) in response to perceived terror threats	C13–14
18 Religion in public life*	C17–21
19 Religiosity and attendance/participation	C22–23
20 Discrimination*	C24–25

Note: *These issue-fields have detailed subcategories.

Examples of how the claims-making database may be used

So far, we have outlined how the categories for the news analysis are derived from the logic on the survey schedule. This means that the data gathered from the news analysis cannot be used to give a representative view of the political climate within a country, but should really only be used in analyses in conjunction with survey analyses. Of course, the point of the database on claims-making is that it is highly flexible and a resource for researchers to use when they examine a research question by analysing the survey data. Such investigations can take many different forms, focusing on specific issues or questions, can address single countries, or apply cross-national comparison. Researchers’ own decisions will shape their specific inquiries. Here we present two straightforward examples to demonstrate how the media data might be used with the survey data to assist in specific research inquiries.

First, we start with a striking finding from the survey data and examine the news data to see whether this can be traced to a specific event, or event cycle in the ‘political climate’ coded from news. This example takes the case of European integration in Ireland. Second, we start from a compelling finding in the coded claims-making data and then turn to the survey data to see whether we find this resonance in ‘political climate’ is present in survey findings. Here the topic is state intervention and its limits and, in particular, a state’s tolerance and recognition of gay/lesbian rights, again in Ireland.

Example 1: European integration (deepening/enlargement) in Ireland

Our first illustrative example starts from a finding in the ESS in Ireland over the issue of Europe. The question in the survey is:

B34: Now thinking about the European Union, some say European unification should go further. Others say it has already gone too far. Using this card what number on the scale best describes your position?

Range 0–10

Where:

00 Unification has already gone too far

10 Unification should go further

On this question there was a marked shift between Survey Round 2 (2004/5), when the mean position was 5.69, and Survey Round 3 (2006/7), when the mean was 4.87. According to this evidence, public opinion in Ireland moved towards the view that European unification had already gone too far, with a shift of 0.82 between the two periods of the survey fieldwork. If we check quickly with Ireland’s near neighbour, the UK, we see that there was also a shift towards the view that union has gone too far, but this was not as pronounced as the change in Ireland (Survey Round 2 mean 4.96; Survey Round 3 mean 4.72; shift 0.22 against European integration), even if it started from a less favourable stance over Europe. These findings raise the question of whether there was something specific occurring within the political climate of Ireland during the period of the survey fieldwork that gave rise to this marked shift against European integration between the two surveys. It leads to a general descriptive question: Was Europe especially salient in the media, over what issue, and what meaning did political actors attribute to European issues? A supplementary question of specific relevance for a researcher using the survey data is: Was the specific salient issue over Europe of such a nature that it might have had a short-term effect on responses in the period when the fieldwork was conducted?

The claims-making database has the flexibility to provide some quick relevant answers to these questions when the researcher runs a query through SPSS. A general picture can be built of the salience of Europe in the Irish news. First, one can answer the question about the salience of the European issue relative to other issues in public debates mobilized by political actors and carried by the Irish news. Here the finding is that political actors’ claims-making over Europe accounted for an 11.6 per cent share of the coded claims in the period of Survey Round 3, making Europe the second most highly ranked salient issue in Ireland. Second, one can ask about the position of the debate over Europe: overall, were there more claims with negative evaluations of European integration, than positive evaluations? A quick indicator comes from each claim being coded for its position over European integration (−1 in favour of, or for more [integration, enlargement, deepening] EU, including EU institutions; −1 against, or for less [integration, enlargement, deepening] EU, including EU institutions; 0 neutral claim about EU, and EU process, without being for or against). The finding is that the mediated political debate in Ireland produces a mean position score of −0.03 over Europe. Although in absolute terms this is a neutral evaluation of European integration, it compares with the overall mean scores for all claims in Ireland of +0.20, which shows that European integration was an issue that was more negatively evaluated than others.

Running queries through the claims-making database, it is possible for the research to move further in determining the actors who were most prominent protagonists in the debate over Europe in Ireland, when it occurred over time (for example, were there specific peaks and troughs?), and what it consisted of substantively. The researcher is able to follow the inquiry by zooming in on more and more detail until arriving at the actual coded details of claims. For example, the Figure 1 shows the distribution of the sample of claims over Europe in Ireland over the relevant coded weeks. Here the darker shading represents the 46 claims over general European integration issues, while the lighter shading shows the 24 claims that referred specifically to the potential immigration consequences of EU enlargement to include Bulgaria and Romania. Here we see confirmation that from week 3 to week 22 of the coverage, the public debate over Europe in Ireland was strongly shaped by this single issue.

OPEN IN VIEWER

Figure 1.

Distribution of claims over European issues by time (week), Ireland.

It is also possible to retrieve actual cases of the claims made, again in order for the researcher to evaluate whether this issue of possible immigration consequences from Bulgaria’s and Romania’s accession to the EU might have shaped the political climate over European integration in a specific way at this time. We list a few examples from the data:

Claim-maker:

Justice Minister Michael McDowell

Claim:

‘Ireland strongly committed to Geneva convention for refugees but Romanians cannot seek asylum after EU membership’

Claim-maker:

Archbishop of Dublin Dr Diarmuid Martin

Claim:

‘Immigrants from Bulgaria and Romania should soon get the same rights as others from the EU’

Claim-maker:

Bertie Ahern

Claim:

‘Ireland received half the number of migrants as Britain from 2004 enlargement even though Britain’s population is 15 times more’

Claim-maker:

Spokeswoman for the employment commissioner EU, Vladimir Spidla

Claim:

‘Restriction on Romania and Bulgaria regrettable as the 2004 enlargement was economically positive’

Claim-maker:

Romanian foreign minister Mihai-Razvan Ungureanu

Claim:

‘Romania to impose reciprocal restrictions on Irish workers if not granted full access after enlargement’

In sum, this example is meant to be illustrative. Ultimately, the decision comes down to the researcher to decide whether on the basis of the available evidence from the news data there could be a bias in the Irish survey responses because of specific salient events that occurred during the period of data collection for the survey. Our aim is simply to provide a methodological tool to arrive at such a decision based on some degree of empirical evidence.

Concluding remarks and discussion

This article documents the methodological steps that were taken in an attempt to use news analysis to provide an indicator for the ‘political climate’ in a way that could aid interpretation of public opinion survey findings, either across time, or across countries. The simple idea applied was to use the selective function of news media in choosing the most relevant political events and issues in a specific country at a specific time to construct ‘news’ in order to determine whether what was happening politically in a country at a given time could potentially have shaped the responses picked up by questions in the survey. Viewed narrowly, the data from news contents analysis on the ‘political climate’ can locate whether exceptional political occurrences at the time of opinion survey fieldwork might account for unexpected survey findings. In this way, the news analysis becomes a control mechanism for survey research on public opinion. This has been the main usage of the ‘tool’ that we developed for the European Social Survey.

Previously, news, the output of professional journalism, had been rarely analysed by survey researchers. Although contextual data were deemed important and relevant when measuring changes in attitude over time, the way events were mediated was not something examined systematically by the ESS. Our aim was to provide a methodological tool for the survey with which to examine political issues and events and this has now been adopted as the required method for country coordinators from Round 6 of the survey.

Based on our experiences, we believe that in the future there is greater scope for interaction between communications researchers and survey researchers, two scientific communities that have remained relatively distinct in the past. Given the shared interests of both communities in unpacking and understanding where ‘public opinion’ comes from, we consider that this can be a fruitful area for collaboration in the future. Indeed, there could be significant gains from running large-scale cross-national projects that monitor the contents of news across time and countries, and which run parallel to survey research, such as the ESS, that monitors changes in public attitudes. The availability of such resources would allow for a whole range of research activities addressing questions about the relationships between media contents and public attitudes on topics. So far, we are not at such a stage of scientific development and the generation of such resources would require a similar type of institutional and financial backing for news analysis in several countries to that received by the ESS from European-level funding.