A game of thrones: Organising and legitimising knowledge through PISA research

The development of ILSA

The International Association for the Evaluation of Educational Achievement (IEA) was the first organisation to focus on large-scale assessments of students’ achievements in the late 1950s and early 1960s (Pettersson, 2014a). The first IEA study differs from other comparative education studies undertaken at that time in that it tried to introduce an empirical approach into a field dominated by cultural analysis (Foshay et al., 1962). The study concluded that cross-national studies of educational performance could produce comparable results, which at that time were startling (Purves, 1987). The IEA continued to develop several assessments influencing both policy and research (Pettersson, 2014a), the most well-known of which are TIMSS, TALIS, CIVED, ICCS (for a further discussion see Lindblad et al., 2015). Eventually, criticism of the IEA emerged with regard to the lack of policy relevance. In response, the OECD started to develop indicators and assessments of their own (Pettersson, 2008).

Since the introduction of PISA 2000 by the OECD, both the programme and its results have been widely disseminated in bureaucratic, political and media contexts (Carvalho, 2009; Grek et al., 2009; Pettersson, 2014a). It is important to acknowledge that PISA is not just a test. In conjunction with the tests, meetings are held to discuss both the tests and the results. The results are then published in reports and disseminated worldwide. A number of actors from various countries are involved in PISA activities; for example, public and private research centres, OECD professionals, policymakers, bureaucrats and technicians, as well as independent researchers using the data or results from PISA to investigate different fields of knowledge. The involved actors have different knowledge, interests and perspectives. From this point of view, PISA is a conglomerate of activities, objects and actors that generates diverse actions in different social spaces at different levels (Carvalho, 2012). This means that the effects of PISA are diverse and not always as linear as they might first appear.

ILSA in policy and research

One of the most obvious connections in discussions about ILSA is that between policy and research. Wagemaker (2014) describes this connection in the following way:

Common conclusion is that stakeholders need to recognize that major policy initiatives or reforms are more likely to result from a wide variety of inputs and influences rather than from a single piece of research. Research is also more likely to provide a heuristic for policy intervention or development rather than being directly linked, in a simple linear fashion, to a particular policy intervention. This outcome is due not only to the competing pressure of interests, ideologies, other information, and institutional constraints, but also to the fact that policies take shape over time through the actions of many policymakers. (Wagemaker, 2014: 12–13)

The research activities generated by ILSA are not always explicit. Even though researchers and policymakers may appear to live in different worlds, there are connections and overlaps.

The expansion of ILSA is well documented (e.g. Kamens and McNeely, 2009), but questions continue to be asked about the extent to which ILSA has an impact on policy and research. The question that is most frequently asked is whether ILSA contributes to educational reforms and improvements in different national settings. The question is understandable, considering that most of the organisations behind ILSA have more or less explicit aims. However, it is also relevant in relation to the vast production of grey-zone literature produced on the impact of policy at different levels (cf. Lindblad et al., 2015). In the Handbook of International Large-Scale Assessment, Wagemaker (2014) summarises the impact of different ILSA tests and identifies seven effects: i) the growth of ILSA; ii) a widening discourse; iii) changes in educational policy; iv) curriculum changes; v) changes in teaching; vi) capacity building and research endeavours; and vii) global and donor responses. What is striking is that the impact of ILSA on research communication is not obvious, but is instead described as sub-categories under certain headings. Although research on this issue is sparse, it does exist. In order to understand the impact of ILSA, we have to understand research communication in terms of how educational concepts, data and results are developed, disseminated and interchanged.

Journals and research fields

The first analysis stage of the corpus mapped the journals and research fields involved. The most referred-to journal is Comparative Education, which is also the second most used journal referring to the core articles. Moreover, the second most referred-to journal by the core articles is the Journal of Education Psychology (also the sixth most referring journal). The journal that refers to the core articles the most is the Journal of Education Policy, which is also the seventh most referred-to journal. In order to investigate the research question further, the research environments in which the journals were active were mapped using the classifications available in WoS of the total number of referred-to and referring articles:

In Table 2, psychology is the primary field for the referred articles. The second most common is educational research. Further, the field of educational research refers more substantially to the core articles than any other field. The field of psychology is the second most referring field. Sociology is also present as a field that is referred to by the core articles and that also refers to them. Business economics, science technology, social sciences and women’s studies, which are not indicated in the table, all appear in the top 10 as referred-to fields. Based on the table, the referred-to and referring journals and the fields of educational research and psychology frame the corpus.

OPEN IN VIEWER

Table 2.

Research fields of the journals.

Field	Percentage of all the articles referred to by the core articles	Field	Percentage of all the articles referring to the core articles
Psychology	34%	Education, educational research	47%
Education, educational research	29%	Psychology	27%
Sociology	12%	Sociology	3%

Main countries

In order to determine the activity of different countries in PISA research, we mapped the affiliation countries in the corpus. This provides information about the cultural frames and possible bias in the PISA research, which are important when explaining how knowledge is established and disseminated.

As Table 3 shows, USA is the primary country. Other countries in the table – Australia, Germany, England, Finland, China, Belgium, Italy, Canada and the Netherlands – appear as referring and referred-to by the core articles. This makes the lists of countries referred to by the core articles and those referring to them highly similar, with 11 out of 15 countries on both lists, thus indicating the supremacy of some countries in the field. It is noteworthy that the referred-to and referring authors mostly come from English-speaking countries, thus providing a bias towards values and ideas connected to these societies.

OPEN IN VIEWER

Table 3.

Country of affiliation of PISA researchers.

Country	Percentage of all the articles referred to by the core articles	Country	Percentage of all the articles referring to the core articles
USA	37%	USA	16%
England	16%	Australia	13%
Australia	15%	Germany	11%
Germany	7%	England	10%

Authors and nodes of PISA research

In order to investigate nodes of research and make claims on epistemic cultures, the authors referred to by the core articles and referring to the core articles were mapped. It became clear that some authors were more used and active than others in establishing nodes of research. The most referred-to authors in the core articles are presented in Table 4, and the authors referring to the core articles in Table 5. In these tables, the 15 most referred-to or referring authors are portrayed in order to provide manageable information and illuminate the major actors in the field.

OPEN IN VIEWER

Table 4.

Authors most referred to by the core articles.

Authors	Times referred by the core articles	Affiliation
Marsh, Herbert W	37	Educational Psychology, Australian Catholic University, Australia
Ludtke, Oliver	12	Leibniz Institute for Science and Mathematics Education, Germany
Trautwein, Ulrich	12	Psychology, Education Sciences, University of Tübingen, Germany
Craven, Rhonda G	10	Institute of Positive Psychology and Education, Australian Catholic University, Australia
Lingard, Bob	10	University of Queensland, School of Education, Brisbane, Australia
Köller, Olaf	7	Educational Research, Leibniz Institute for Science and Mathematics Education, Germany
Preacher, Kristopher	7	Quantitative Methods, Vanderbilt University, USA
Baumert, Jürgen	6	Max Planck Institute, Berlin, Germany
Rawolle, Shaun	6	Faculty of Arts and Education, Deakin University, Australia
Eccles, Jacquelynne S	5	Institute for Social Research, University of California, USA
Gorard, Stephen	5	Academy of Social Sciences, Durham University, Great Britain
Hau, Kit-Tai	5	Department of Educational Psychology, Chinese University of Hong Kong, China
Hyde, Janet S	5	Department of Psychology and Department of Gender and Women’s Studies, University of Wisconsin, USA
Robitzsch, Alexander	5	Bundesinstitut für Bildungsforschung, Innovation und Entwicklung des Österreichischen Schülwesen, Austria
Seaton, Marjorie	5	Institute for Positive Psychology and Education, Australian Catholic University, Australia

OPEN IN VIEWER

Table 5.

Authors who refer most to the core articles.

Authors	Times referring to the core articles	Affiliation
Lingard, Bob	23	University of Queensland, School of Education, Australia
Sellars, Sam	22	University of Queensland, School of Education, Australia
Marsh, Herbert W	20	Educational Psychology, Australian Catholic University, Australia
Morin, Alexandre JS	14	Institute for Positive Psychology and Education, Australian Catholic University, Australia
Takayama, Keita	14	University of New England, Australia
Nagengast, Benjamin	12	University of Tübingen, Germany
Parker, Philip D	10	Institute for Positive Psychology and Education, Australian Catholic University, Australia
Abduljabbar, Adel Salah	9	King Saud University, Saudi Arabia
Alegre, Miquel Angel	8	Catalan Institute of Public Policy Evaluation, Spain
Preckel, Franzis	8	University of Trier, Germany
Abdelfattah, Faisal	7	King Saud University, Saudi Arabia
Benito, Ricard	7	Autonomous University of Barcelona, Spain
Liou, Pey-Yan	7	National Central University, Taiwan
Feniger, Yariv	6	Ben Gurion University, Israel
Hyde, Janet S	6	Department of Psychology and Department of Gender and Women’s Studies, University of Wisconsin, USA

Tables 4 and 5 show that Lingard, Sellars and Marsh refer to the core articles the most. Marsh is also the most referred-to author by the core articles (Table 4). Moreover, Craven, Seaton and Rawolle are referred (Table 4) authors and are also authors of the core articles. Takayama and Nagengast refer (Table 4) to the core articles and to the authors of the core articles. Taken together, this indicates an overlap between the referred-to authors, the authors of the core articles and the referring authors. This observation confirms that some epistemic cultures seem to be more important than others in framing PISA research.

Nodes of research

The authors Lingard and Marsh are referred to by the core articles, are authors of core articles and refer to the core articles. Conclusively, Lingard and Marsh can be regarded as important establishers of research nodes in the investigated selection of PISA research. Marsh is a psychologist from Australia and in his role as a professor at the University of Oxford has had many articles published on, for example, PISA results. When mapping Marsh’s co-writers, not only in relation to the corpus, many of the other authors listed in the Tables 4 and 5 appear; for example, Morin, Nagengast, Craven, Parker, Lütke, Trautwein, Seaton, Abduljabbar and Köller. Even though many of Marsh’s co-writers are Australian, he also has an extensive cooperation with researchers in England and Germany. Further, Marsh publishes mostly in the fields of psychology and the social sciences, and primarily in the Journal of Educational Psychology.

Another node of research is concentrated around Lingard, an educationalist from Australia, who has a global network and is mostly published in policy journals. Two of Lingard’s co-writers also appear in Tables 4 and 5, namely Sellars and Rawolle. Another important co-writer is Grek – one of the authors of a core article selected for this study. In relation to Marsh, there is a major difference in terms of scientific field, in that most of Lingard’s contributions relate to the social sciences. Lingard also has an extensive cooperation with researchers in different countries in the English-speaking world. Most of these articles are published in the Journal of Education Policy.

By combining the retrospective and prospective analyses, we have been able to highlight some of the characteristics of a part of PISA research. These features are illustrated in Table 6.

OPEN IN VIEWER

Table 6.

Main findings.

	Findings
Field of research	A relation between the field of psychology and the field of education
Countries	English-speaking and westernised countries are the major contributors of PISA research
Authors	Two nodes of research, one within the field of psychology concentrated around Marsh, and one in the field of educational policy centred around Lingard

In this selection of PISA research, some authors tend to be more visible than others. Two especially important nodes of PISA research are highlighted, both of which are located in English-speaking countries. One node relates to psychology and the other to education. A relation between the field of psychology and the field of education has also been identified. These findings are discussed in the next section.

A relation between psychology and education

One of the study’s major findings relates to the connection between the field of psychology and that of education. Thirty-four percent of the articles referred to by the core articles represent the field of psychology, and 29 percent the field of educational research, which indicates that psychology is the dominant field. It is important to remember that by its very nature, WoS may prioritise psychology rather than education, which means that the result could be biased due to software construction. The situation is reversed when it comes to articles referring to the core articles. Here, 47 percent of the articles referring to the core articles are assigned to the research field of education and 27 percent to the field of psychology. This could indicate, in relation to the limitation concerning WoS, that WoS may not be overly biased in favour of psychology. In order to explore this, a more detailed database analysis would have to be performed.

The third significant research field to emerge in the analysis is sociology, for both referred-to and referring articles, with a third place on both lists. Publishing patterns for the less referred-to and referring research fields are more scattered. It is noteworthy that the corpus manifests in the field of psychology in terms of inspiration (retrospective analysis) and disseminates to the field of education (prospective analysis). These findings indicate a relation between the epistemic cultures of psychology and education in the construction of PISA research. We argue that PISA research is situated in-between these two scientific cultures. In this, based on the publications referred to by the core articles, it is possible that psychology has a slightly more predominant role in the construction of the field. It can also be conjectured that in the publications referring to the core articles, education becomes increasingly more dominant in the field of PISA research. The findings also indicate that among those who refer to PISA research, educational research is important. The question then arises as to whether the status of the field of psychology and the knowledge production in psychology are used to gain legitimacy within the field of education.

English-language-framed PISA research

As might be expected, the authors of the articles in the corpus are primarily based in English-speaking and westernised countries. The US has 37 percent of the referred-to authors and 16 percent of the referring authors. The next highest countries in these categories are England (referred-to authors 16 percent) and Australia (referring authors 13 percent). China is the only non-westernised country to be included in both the lists of referred-to and referring countries. This result can be interpreted in different ways, although the favouring of authors from English-speaking countries is clear and underlines the influential position of English-language research journals. As such, the result is perhaps not particularly striking, although the bias in favour of English literature and westernised countries within the field is significant, especially if language and language-dependent concepts are considered as important for how knowledge is created and legitimised. A result of these observations is that the establishment and development of knowledge in these specific epistemic cultures is highly biased in favour of English concepts and is also framed in that particular language. Field-specific concepts are, therefore, framed by the language. A domination of western ideas can also be highlighted and discussed. It is fair to say that the research field is rather narrow, even though PISA is regarded as international. This narrowness is further strengthened by the fact that the PISA assessment itself is normally presented in English. What is not specifically investigated in this study is PISA research presented in languages other than English. Instead, the study is framed by research embedded in WoS, which is biased in favour of English-language journals. Elaborating on non-English-language research could be an interesting study to perform, given that research has shown that when PISA concepts meet national contexts, translations take place that give PISA new directions and new foci. However, further studies would need to be conducted in order to trace the trajectories of PISA reasoning in different languages and contexts.

Nodes of research: two dominant epistemic cultures

By exploring Marsh and Lingard and their co-authors, it is possible to determine the construction and legitimisation of knowledge. This appears in the two nodes of research identified in the bibliometric analysis: one situated within educational policy thinking and one active within the field of psychology. The two presented authors appear as referred-to authors, authors of the core articles and as authors referring to the core articles. It is also obvious that co-writers, especially Marsh and Lingard, are referred to by the core articles, are authors of the articles in the corpus and are referred to by the corpus. This can be discussed in the light of Knorr Cetina and Cicourel (1981), who argue that science is largely a socially organised cognitive activity made up of various epistemic cultures. Marsh and Lingard seem to be important in the organisation of cognitive activity and thereby function as epicentres of two different epistemic cultures of the investigated PISA research and are highly quoted, which is important for the accountability and legitimation of the presented arguments and knowledge (e.g. Kandlbinder, 2015).

In a situation in which the most central articles of PISA research tend to represent a somewhat narrow field in terms of authors, there is also a risk that the field will become self-authorising. Self-authorising means that author A says that x is a scientific fact by referring to author B and that x is, therefore, presented as an evidence-based fact. This makes us wonder whether self-authorising over-emphasises specific reasoning within the field and, as such, is self-generating and self-referential, as Hacking (1992) pointed out in his important work. This question is prompted by the overlapping that is observed in the studied PISA research, as exemplified in the analysis of the corpus. For example, authors are found in the list of referred-to articles and as authors of the referring articles. The author of the 10th most referred-to article in the corpus is also fifth on the list of authors referring to the corpus. Thus, there is a tendency for PISA research to be constructed and managed by a clique of researchers in dominant positions with regard to the volume and impact of their publications and the opportunities to guide the narrative surrounding the scientific debate about PISA. In our analyses, two important nodes for PISA research are identified. Both these nodes are well established, with international cooperation, large productivity and numerous co-writers further establishing their centrality in PISA research. This could indicate the importance of PISA research for capacity building, as previously described in relation to ILSA tests (see Wagemaker, 2014). We argue that this capacity building includes the legitimisation of knowledge and actors through nodes of research constructed by self-authorising and self-referential publishing patterns and co-authorship created in specifically demarcated epistemic cultures.

It is noteworthy that when it comes to inspirational authors (retrospective analysis) and dissemination in the field of education (prospective analysis), the PISA corpus is very similar. This may not be all that surprising, considering that scientific authors tend to promote their own careers and positions by referring to themselves and their colleagues/collaborative partners for research accountability. This trend is perhaps also acceptable, given that it is normal to think about your own work as important for answering different research questions and creating new ones. Individual researchers’ work can be interpreted as work in progress, where previous work always plays an important role. In our view, this system of individual researchers’ self-referential is perfectly normal in contemporary science, but can lead to problems when it comes to the validity and reliability of individual works and a research field at large. This has to do with the system of accountability, as well as intellectual reasons of positioning. As such, we wonder whether the system of accountability actually fuels the construction of nodes of research in a race for legitimacy, which in itself creates two dominant epistemic cultures.

The characteristics of PISA research

It is interesting to reflect on Wagemaker’s (2014) statement that one impact of ILSA is a widening of discourse. Based on our findings, we illuminate the fact that, in comparison to many other scientific fields, very few articles and actors are connected to the two nodes of research dominating the scope of PISA research in scientific peer-reviewed journals. Moreover, this also indicates that many research articles rarely disseminate as references, which leads to speculations as to how wide the PISA discourse really is. Yet, the fewta articles that are produced have a disproportionate impact on the creation of a specific PISA reasoning. This can also be discussed in relation to the dominance of English as the language of publication, which may mean that concepts are discussed differently.

This study has discussed the role that scientific articles play in the creation of knowledge structures (Börner et al., 2003) in a specific section of ILSA research. As PISA tests and PISA research are constantly used in arguments for reform initiatives and curriculum development, it is important to study and analyse this specific knowledge field in order to understand how such knowledge is organised and legitimised. This ought to lead to a better understanding of how notions such as knowledge, curriculum, assessment and evaluation are discursively formed inside and outside a scientific community and how different research fields function as interfaces, where intellectual yielding occurs and academic capital is interchanged. The findings suggest an interchange of legitimisation in which the USA and other English-speaking countries set the scene for academic discourse and transformation. Therefore, PISA knowledge seems to be organised in publication patterns that favour the English language and the field of education. Moreover, it could be that authors draw on knowledge constructions in psychology to gain legitimacy within the field of education. An indication of this is the publication pattern in which psychology is a major field for the referred to authors, but where the field of education is becoming more important through the authors of the core articles and those who refer to them. How this relation between psychology to education changes or affects the field of education and vice versa is an interesting question that is worthy of further research.

Two further important nodes of PISA research have been located; one dominated by psychology and the other by the educational sciences. The two research nodes function as epicentres in different epistemic cultures of the publishing patterns, in that the authors and co-authors refer to and reference each other’s work to the extent that only a few actors control what is formulated in terms of PISA knowledge. Both nodes establish thrones in PISA research, where the interchange of legitimacy is conducted through the cultivation of each other’s knowledge. This development can be discussed in terms of assortativity and preferential attachment. Assortativity is the tendency to connect to actors who are similar (Guns, 2014), whereas preferential attachment is the tendency to connect to successful actors (cf. Barabási and Albert, 1999). Hence, the corpus appears as a self-referential and self-authorising activity. As such, the study presents two nodes, which can be characterised as two different epistemic cultures, that can be described as two ‘thrones’ in PISA research on which PISA discourses are constructed. Moreover, we wonder whether research accountability leads to what we metaphorically express, in line with the well-known George RR Martin novels and HBO series Game of Thrones. The reasons for this are, just as Merton (1973) stated, that rivalry occurs within the field concerning who the actors are to constitute knowledge. The rivalry can be seen by using the theoretical concept of epistemic cultures for understanding that there are different groups or ‘houses’ that all claim supremacy on how to understand specific concepts and claims within the field as ‘facts’ and ‘truths’. The ‘thrones’ then establish specific epistemic cultures. As a result of this, we believe that the metaphor of game of thrones is suitable for our findings because, just as in the series and novels, there is rivalry going on within the scientific field concerning how and on what grounds ‘facts’ and ‘truths’ are constructed, as a continuing process with no obvious winner.