Effects of international assessments in education – a multidisciplinary review

The extensive use of quantified data in decision-making is a widely observed trend in various contexts (Power, 1999). Advanced electronic data analysis techniques, available and encompassing data, and facilitated global exchange of information have led to a steady growth of assessments that are used to make informed policy decisions. The appealing added value of assessments is that they give an easily accessible account of cause-effect relations – although, in consequence, these relations are very often spurious if the study design or the respective data analysis do not account for the effects of unobserved characteristics. In this regard, it is often believed that the analysis of assessment data can reveal solutions to identified problems – in the sense of best practice examples.

In line with the general trend, international assessments in education have also become a global and accepted phenomenon (Heyneman and Lee, 2014). Today, they are a well-established instrument to compare and evaluate the performance of students, schools or national education systems. PISA, TIMSS, PIRLS, AHELO, PIAAC are some of the best known abbreviations of these studies. These assessments sometimes result in elaborated rankings and ratings that are used to increase accountability and are meant to guide politicians or educators (Feniger and Lefstein, 2014). This recent phenomenon was also comprehensively covered by the European Educational Research Journal (EERJ). The new trend of quantitative education assessments and its repercussions on national education systems have been analyzed in several EERJ special issues and stand-alone papers. ¹

The starting point in assembling the contributions of this special issue is that we believe we need a more broadened approach than that which has been developed in the community of research thus far to understand the diverse impacts of international education assessments. We therefore gather contributions in this special issue which make use of diverse – and to some respect – unusual lenses in order to understand how international education assessments have some grave and far-reaching implications for the organization and structure of national education systems. For instance, these studies sometimes involve new international actors, like international organizations, as influential and partially autonomous drivers for reform processes in education (Grek 2010; Martens et al., 2007). On the one hand, these new actors take over some tasks that states (and national actors) used to deal with. By pooling resources they are able to act more efficiently than states on their own. In this regard, it is rational for states to delegate educational assessments to, for example, international organizations (Martens and Wolf, 2009). On the other hand, the increased importance of new actors in international education puts states (and sub-state actors) under constant pressure to operate at the international level, to justify their national decisions against the background of international assessments, and to also transfer some of the leadership regarding education ideas and ideals by providing a platform for mutual exchange (Carvalho, 2012).

Perhaps most prominently, the PISA study of the OECD has become an international venture with now more than 70 countries and regions participating in the triennial comparison of national education systems. However, international assessments in education are not a new development; the First International Mathematics Study (FIMS) conducted by the International Association for the Evaluation of Educational Achievement (IEA) dates back to 1964. However, the last decade has seen a significant growth in media attention as well as political responses to such studies. Not all surveys attract the same attention as PISA, but many spark lively debates about the purpose of education and its efficient realization. With their simultaneous analysis of performance and its determinants at the individual, school, and system level, assessments also establish a specific interpretation of ‘best practice’. PISA, for example, was administered explicitly in order to guide educational reforms (Anderson et al., 2010).

In fact, educational reform as a direct response to results of international assessments can be observed in many countries and has been analyzed by many different scholars from different scientific disciplines (Breakspear, 2012; Knodel et al., 2010; Lingard and Grek, 2006; Martens et al., 2010). While there is considerable research devoted to how PISA influences national education systems, most of the recent literature looks only at the level of formal policies, without taking mechanisms and effects on actual practices and outcomes into account (Teltemann and Klieme, 2016). Further papers focus on the ‘nature’ of PISA and the ways it is produced and consumed (Mangez and Hilgers, 2012; Ozga, 2012), as well as on more general trends in educational governance (Lussi Borer and Lawn, 2013; Williamson, 2016). However, there is still a lack of solid findings on the interplay between international assessments, educational reform, practices and learning outcomes. This observation motivates this special issue on the long-term effects of international assessments in education. We are particularly interested in analyzing how PISA and other assessments permeated the understanding of education and how international assessments shape policy-making.

Before analyzing their actual impact we have to elucidate how international assessments in education have become so powerful. Inspired mostly by neoliberal approaches to autonomy and accountability, international assessments are viewed as instruments which contribute to new forms of governance in education (Grek, 2009; Grek and Ozga, 2009; Meyer and Benavot, 2013). In accordance with the more general idea of globalization and transnationalism, they substantially shape the understanding of ‘good education’ and define how academic performance is assessed today. From the perspective of the Stanford School of Institutionalism, assessments, as well as a convergence of education systems that is promoted by international organizations, are only a concomitant feature of a world polity (Meyer et al., 1992; Powell and DiMaggio, 1991). While the rise of assessments in education might seem a ‘normal’ or ‘natural’ phenomenon from these theoretical perspectives, they are also highly controversial for some scholars and stakeholders. The increasing focus on international standardized assessments has been criticized sharply, in particular for its strong focus on competencies, for its simplistic presentations of results and for methodological weaknesses (Connell, 2013; Dohn, 2007; Goldstein, 2004; Prais, 2004; Uljens, 2007).

Moreover, the increased involvement of international organizations and private actors in testing raises questions about democratic legitimization and ideological biases. This is seen to be particularly critical if international organizations make use of these studies to pursue their own policy agenda. International assessments are applied from outside to evaluate domestic education systems and shape policy discourses, sometimes without being backed up by the consent of those being tested. At the same time, however, international assessments in education provide rich databases, give new impulses, and have triggered extensive research in various disciplines, thereby enhancing our knowledge of ‘what works in education’ (Dronkers 2010; Hanushek and Woessmann, 2006; Hanushek et al., 2013; Van de Werfhorst and Mijs, 2010).

There is little doubt that international assessments established a ‘new centre of gravity’ in the field of education and that they re-shaped education policy-making and practices in many countries. However, the question we address in this special issue is how this new mode of governance affects the ways education is defined and learning is organized today. We seek to explore the dynamics and repercussions of international assessments in education from a multidisciplinary perspective. In contrast to previous research on the effects of international assessments, we do not look at formal policies in the first place, but rather on less obvious consequences of PISA and comparable projects. We assume that international education assessments reach beyond direct policy changes and, at least in the long run, also influence how educational quality, educational practice and the meaning of empirical data is defined.

The contributions of this special issue point out that there is good evidence for a new mode of education governance promoted by large-scale comparative studies. Furthermore, they also point out that the effects of this new mode are evident in diverse areas of the education system and are not restricted to the dimension of direct policy-making. The special issue also shows that the breadth of effects of international assessments can hardly be analyzed from the perspective of only one academic discipline. The various contributions are rooted in different academic disciplines and draw a complementary picture of the complex consequences of assessment in education. We received 15 papers, of which seven have been selected for external review. After review and revisions, five papers have been chosen for publication in this special issue.

In the first paper, Sigrid Hartong analyzes the effects of international education assessments, in particular PISA, as a new constellation of (big) data mediation from a sociological perspective of digitalized governmentality (see also Williamson, 2016). She demonstrates that not only the assessment itself transforms education systems, but also that the assessments are interlinked with the general processes of educational understanding within a society. Data processing and analysis in educational large-scale assessments takes place beyond formal decision-making structures, as Hartong argues, operating as ‘hidden’ processes. New actors and networks of transnational actors have emerged who design and conduct international assessments. Think tanks, business firms and industrial networks act as mediating actors between the global and the national level. They become increasingly involved in designing and using educational big data information technologies and applying them in order to influence school practices, administrative organization and learning techniques. In sum, Hartong points out that by producing a huge amount of data, international education assessments permeate national policy-making structures. Actors producing such big data have become more powerful in influencing not only political decisions but also making an impact on educational practices.

The second article by Mike Zapp focuses on Germany and traces processes of institutionalization in the field of empirical education research. Reaching far beyond mere effects of formal policies, Germany experienced the development of a new organizational field after the ‘shocks’ of TIMSS and PISA in the late 1990s and early 2000s. A crucial event was the foundation of the Empirical Educational Research Association as a competitor to the traditional German Educational Research Association. The author draws on neoinstitutionalist literature and on the concept of organizational fields, and shows how the various building blocks of the new field are constructed by ‘institutional work’ and ‘institutional agency’. Three key features characterize the organizational field of empirical educational research in Germany: reliance on data-intensive empirical analysis, internationality, and focus on practical applicability. This puts empirical educational research in Germany in the position of a ‘national node’ in a new international research landscape, which contributes to changes in the production of educational knowledge.

The effects of international education assessments are not restricted to the OECD-world but also have a more global reach. In his contribution on quality assessment in Brazilian basic education Jaakko Kauko shows that a culture of educational evaluation was well established in Brazil. This development was promoted by diverse domestic actors, by referring to international role models, and by cooperating with international organizations. From a historical perspective, the author traces the steadily growing importance of quality assurance and evaluation for Brazilian education policy and analyzes how quality evaluation became a significant feature in the Brazilian education sector over the course of almost a century. Kauko shows that the development of assessments in Brazil was strongly influenced by international standards derived from international organizations and other states. In particular, the US and the World Bank (and later the OECD) were prominent players in shaping Brazilian quality assurance policies in the field of education. In this context, Kauko also argues that the transformation processes can be understood from the perspective of sociological institutionalism, with its emphasis on spreading a world culture. Brazil was influenced by a converging international discourse on educational governance and standardized testing. Domestic politics has played a decisive role in picking up international ideas and diffusing them at the national level. Hence, the emergence of quality assessment in Brazilian education can best be understood as an interplay of the national and the international level – with domestic actors as the main driving forces for bringing international actors in.

The article ‘Tracing French policy PISA debate: A policy configuration approach’ by Xavier Pons analyzes the development of the French debate on PISA between 2001 and 2014. It argues that PISA’s impact on policies can be characterized primarily as an instrument for legitimizing already existing policy proposals instead of triggering new debates and solutions to the French education situation. PISA thus increasingly became a communications tool and part of the policy narratives in official speeches. France, in fact, is an interesting case for analyses as one would not expect an internationally comparative assessment to find consideration at all in national political debates. In particular, because of its regular claims of cultural exceptionalism, France is often regarded as being more ‘resistant’ to globalization and global discourses, and this is also true for the education sector. Pons therefore shows the particular French ‘vernacular’ – as he calls it – with which national debates absorb international discourses.

Radhika Gorur’s contribution, ‘Seeing like PISA: A cautionary tale about the performativity of international assessments’ looks at PISA from a literary approach. Using James Scott’s study of 18th-century German forestry practices as a parable, the author’s aim is to raise our awareness of how PISA can foster a dangerous development towards monoculturality in education policies. Just as the rapid growth of forestry practices in Prussia and Saxony led to massive destruction in the long run, the unquestioned belief in international comparative evaluations, worldwide standardized measurement and global education metrics may destroy diversity in education. Gorur’s point is that PISA does therefore not only produce descriptive data, but becomes a performative tool, which can have implications, perhaps damaging ones, in the long run. In essence, the author argues that states are, therefore, now ‘seeing like PISA’, which means they are analyzing, evaluating and reforming their education systems with PISA benchmarks and data in mind. PISA therefore is a significant example of the current trend towards education policy-making aided by international indicators and large-scale comparative assessments.

One of the aims of this special issue is to broaden the perspective for a better understanding of the phenomenon of international education assessments. In sum, the application of different disciplinary lenses through the contributions in this special issue develops a more comprehensive picture of research findings, institutional and organizational changes, scientific reactions that can be traced back to international assessments and across different disciplines. They show that the effects of international assessment go beyond formal policy statements. Taken together, the articles of this special issue also demonstrate that international assessments inform how education and education performance are framed. They make clear that international assessments as a tool of educational governance are a global phenomenon. The papers also show that the impact of international assessments manifests itself in formal and direct (‘top-down’) ways as well as in more subtle, hidden ‘roundabout’ ways.

However, although this special issue sheds further light on the manifold changes that came along with the rise of international assessments in education, it is still difficult to disentangle general trends and developments in educational governance from direct effects of assessment projects on education. Educational reform is a continuous, steady project since educational institutions are an obvious place for tackling old and new social problems, such as social mobility or immigrant integration. As education is one of the crucial resources for the sustainability of modern, knowledge-based societies, it is inevitably a contested field. Until now, research on the effects of international assessments on policies and practice, and research on the effects of policies and practice on outcomes have been more or less unconnected. Bringing together these two perspectives would help to judge the ‘real’ effects of assessments – and might also reveal the (merely emphasized) positive effects of international testing projects for reaching common goals, such as equality of opportunity.