PISA,global reference societies,and policy borrowing: The promises and pitfalls of ‘academic resilience’

Introduction

The Programme in International Student Assessment (PISA) measures the reading, mathematics, and science literacy of 15-year-old students around the world. This triennial benchmark measure, administered by the Organisation for Economic Cooperation and Development (OECD), has steadily grown in educational policy relevance since it was first introduced in 2000 in 28 OECD countries and four partner countries. The global reach of this measure is partially evidenced by the participation of nearly 80 nations in the most recent 2018 survey. Additionally, cross-national analyses of the policy impact of different international achievement measures have clearly suggested PISA has assumed priority status and become the most prominent international achievement survey used by governments around the world (Grek, 2009; Meyer and Benavot, 2013; Neimann and Martens, 2018; Volante, 2018). Indeed, research suggests that many industrialized nations have focused almost exclusively on the policy implications stemming from PISA and have largely ignored the results of other notable and sometimes older measures, such as those administered by the International Association for the Evaluation of Educational Achievement (IEA) (Volante, 2016).

Research consistently underscores how policymakers around the world turn to PISA results to help direct their own national reform agendas (Chung, 2019; Pons, 2017). Not surprisingly, the level of reactivity to this measure does vary substantially across international jurisdictions—a result supported by both OECD commissioned research (Breakspear, 2012; Organisation for Economic Cooperation and Development, 2015) as well as external researchers who have examined the utilization of PISA results in policy spheres (Pons, 2017; Volante, 2018). Nevertheless, it is worth noting that the OECD does encourage participating nations to make direct use of PISA results and suggest low achieving countries may benefit from “borrowing policies” from high achieving nations in the hopes of emulating their success. These high achieving nations, such as Finland, Korea, Singapore, Canada, and more recently Estonia, are touted as superpowers and referred to as global reference societies in the academic literature (Volante, 2018). Global reference societies are often lauded for their high achievement and equity—the two defining features of excellence in education (OECD, 2015).

More recently, the OECD has devoted a great deal of energy in promoting the notion of “academic resilience” —which refers to the capacity of individuals to prosper despite encountering adverse circumstances (Agasisti et al., 2018). Countries are compared and contrasted in relation to the relative share of disadvantaged students that are able to achieve at a certain level on PISA, with associations drawn to school-level factors and resulting implications drawn for policy reform. Collectively, policymakers as well as academics have become increasingly concerned with understanding the factors and system features that promote and/or hinder the success of disadvantaged student populations.

This paper examines the promise and pitfalls associated with growing influence of cross-national comparisons of academic resilience, an “eye-catching” but ill-defined slogan. Although we do not discount the value of comparative analyses of student achievement and equity results, as evidenced by our own research (Volante et al., 2019a), our analysis underscores how the OECD’s notion of “academic resilience,” which has come to dominate transnational policy debates, is quite narrow and limited by the measures it uses to assess student competencies. Overall, our discussion examines the contested role of the OECD in the global education policy sphere and highlights the necessity of broader notions of academic resilience for socio-economically disadvantaged student groups. Further, we confront some of the inherent risks of policy development processes that are based on the “tails of the distribution”.

Achievement, equity, and the allure of global reference societies

The OECD considers school systems more or less equitable depending on their ability to weaken the link between individual circumstances and education outcomes. For the most part, they have concentrated their efforts on examining individual circumstances that are associated with achievement gaps between high and low socio-economic status (SES) students, boys and girls, and immigrants and non-immigrants. These performance differences are routinely reported in their numerous online reports and policy briefs. Indeed, a quick key word search on the OECD’s homepage (as of March, 2021), using the word string “equity in education,” returned no less than 26,900 results. Similarly, a significant proportion of the popular PISA in Focus policy brief series—which currently number 110 in total—explicitly contain the words “equity,” “equitable,” “equal,” “inequality,” “inequalities,” “disadvantaged students,” and/or “socio-economic disadvantage” in their titles. Collectively, the policy diffusion approaches of the OECD, which have been characterized as a soft mode of governance, since their policy recommendations are not based on binding legislative mandates, have been largely successful in shifting the gaze of governments to attend to equity issues, in addition to overall achievement results (Volante, 2018).

As previously noted, closing achievement gaps between high and low SES student populations is a key element of system effectiveness, according to the OECD. The OECD has also drawn direct associations between how well countries address these performance differences and their ability to promote social mobility (Organisation for Economic Cooperation and Development, 2018a). The latter underscores the dominant OECD narrative that select countries are better positioned to take full advantage of their human capital (Volante et al., 2017). It is worth noting that the latter position is also supported by other notable International Organizations (IOs) such as the World Bank, which publishes a Human Capital Index that explicitly draws on PISA results (see https://www.worldbank.org/en/publication/human-capital#Index).

The OECD has consistently indicated that “there is no country in the world that can yet claim to have entirely eliminated socio-economic inequalities in education” (Organisation for Economic Cooperation and Development, 2018a, p. 22). At the same time, it is equally clear, that some countries have done a much better job of addressing the performance disadvantages of lower SES student groups. For example, the most recent administration of PISA in 2018 suggested that one in ten low SES students was able to score in the top quarter of reading performance in their countries (known as academic resilience). In Australia, Canada, Estonia, Hong Kong (China), Ireland, Macao (China), and the United Kingdom, all of which score above the OECD average, more than 13% of disadvantaged students were academically resilient (Organisation for Economic Cooperation and Development, 2018b). When one considers the overall mean scores of students in Estonia, Canada, Ireland, Macao, and Hong Kong (China), in addition to their academic resiliency results, it is easy to understand how these countries have garnered a great deal of popular media attention and been promoted as global education leaders. Some have even likened participating countries to superpowers (Butrymowicz, 2016; Coughlan, 2017) —spurring some nations to send delegations to study these high performing jurisdictions in the hopes of replicating their success.

Programme in international student assessment and policy borrowing

The OECD contends that the PISA survey allows educational jurisdictions to evaluate education systems worldwide and provides valuable information to participating countries/economies so they are able to ‘‘set policy targets against measurable goals achieved by other education systems, and learn from policies and practices applied elsewhere’’ (Organisation for Economic Cooperation and Development, 2014, p. 2). There is a clear assumption that lower performing systems can borrow policies and transfer them into their national context to help improve their education systems’ performance. This general notion of policy borrowing, lending, and transfer has always existed in the field of comparative education, but has increasingly become the focus of contemporary research on the policy impacts of PISA (see Davis et al., 2018; Phillips and Ochs, 2003; Steiner-Khamsi and Waldow, 2018; Tan, 2016; Waldow et al., 2014; You and Morris, 2016).

Although the policy borrowing literature has identified both intended positive as well as unintended negative outcomes associated with policy borrowing processes, the academic community has largely cast a negative light on the undue influence of the OECD, and PISA in particular, in the field of education. Perhaps the most public and widespread condemnation of PISA occurred in 2014, when a group of more than 80 prominent academics from around the world penned a critical open letter to Andreas Schleicher, the current OECD Director for Education and Skills. The letter “OECD and PISA tests are damaging education worldwide-academics” appeared in the Guardian—a British daily newspaper—outlined a plethora of deficits and concerns associated with PISA and called for an immediate halt to the next round of testing (Andrews et al., 2014). The main points were reiterated and published a second time in the Policy Futures in Education journal, where the list of signatories grew from the initial 80 to more than 130 as of May 2014 (Meyer and Zahedi, 2014).

One of the chief concerns expressed by academics in the Guardian letter, as well as other scholars which have studied the policy effects of PISA, is that it can lead governments to focus on short-term and simplistic approaches to educational policy reform. Research does suggest policymakers often feel compelled to adopt specific assessment, curriculum, pedagogical, and/or governance policies that feature in high-performing systems (Auld and Morris, 2016; Meyer and Benavot, 2013; Tan, 2016; Volante, 2018). Unfortunately, the research is equally clear that even education policies associated with achievement gaps between low and high SES student populations, such as early tracking, can have different effects across countries (Lavrijsen and Nicaise, 2015; Le Donné, 2014; Parker et al., 2018; Schlicht et al., 2010). For example, Germany, a country that in some Lander tracks children earlier than in any other European country, still managed to decrease educational inequalities due largely to the introduction of a number of complementary education policies that balanced out the impact of tracking (Volante et al., 2019b). Thus, it is the constellation of policies and their interaction within an education system that ultimately influences education outcomes. From this perspective, it is clear policy borrowing is a complex process which must account for a myriad of national education issues (Volante, 2023). Indeed, there are inherent risks in focusing policy reform efforts on distant education systems that share very little, if anything, in common with the historical, cultural, and political milieu of host societies.

Policy reform from the tails of the distribution

One may naturally wonder what is the appropriate “unit of analysis” when attempting to draw conclusions on the effectiveness of education policies and practices for policy learning purposes. Certainly, the field of comparative education is by design oriented toward between country differences. The latter is in keeping with the origins of the field and represented in the foundational 1817 publication of Marc-Antoine Jullien de Paris’ “Esquisse et vues préliminaires d’un ouvrage sur l’éducation compare” (Outline and preliminary views of a work on comparative education) which studied education systems across Europe in the hopes of informing policy reforms within this continent. Not surprisingly, the contemporary practice of policy borrowing, transfer, and learning follows in this positivistic tradition by attempting to compare achievement scores and identify exceptional jurisdictions. Yet, as already noted, countries can vary significantly in relation to historical, cultural, political, and other relevant dimensions.

Perhaps the undue attention and preoccupation that policymakers have shown with reveling in the tails of achievement distributions – particularly reference societies that are vastly different in cultural composition to their own country are short-sighted. For example, is it reasonable for a country such as Canada, with one of the most heterogenous student populations, to draw lessons from Finland, which possesses one of the most homogenous student populations in the Western world? Similarly, should an Asian country like Japan be drawing on “Finnish lessons” in attempting to replicate their early PISA success, given the notable cultural differences that exist between these two countries? These patterns, along with others, have been repeatedly noted as a cause for concern in the literature (see Auld and Morris, 2016; Davis et al., 2018; Dixon et al., 2013; Kamens, 2013; Takayama, 2009; Volante, 2023; Watanabe, 2005) and underscore a rather simple axiom that exists in the field of international achievement testing; namely, the greater the historical, cultural, and political differences that exist across two countries, the more likely the associated policy lessens will lose relevance and applicability. The latter is not meant to be an indictment of comparative education, rather, an important caveat that should inform how policymakers consider and utilize “exceptional cases” for informing policy development.

Interestingly, one of the most powerful and often overlooked ways to learn from exceptional cases—typically referred to as “pockets of success” —rests within, not outside, of countries. Consider decentralized education systems such as those in Canada, Germany, Australia, among others. In such national contexts, there are bound to be regions that have enjoyed different levels of success on international measures such as PISA. For example, the most recent administration of PISA in 2018 indicated that Canadian provinces such as Saskatchewan possessed scores well below the Canadian average in all three tested domains with counterparts such as Alberta in science, British Columbia in reading, and Quebec in mathematics, scoring 45, 40, and 60 standard scores points, respectively, above this province (Council of Ministers of Education, Canada, 2019). When one considers how a score point difference of 30 is roughly equivalent to one grade level, it is easy to see how these achievement differences are significant. Similarly, given the shared national context, the policy lessons garnered across provincial jurisdictions in Canada, Lander in Germany, and states/territories in Australia are likely to be more robust than those that migrate across oceans and involve more distinct student populations. Ultimately, policymakers need to use a much healthier level of skepticism and scrutiny when attempting to replicate the success of global reference societies. Perhaps more importantly, is the recognition that our current global context requires notions of academic resilience that are broader in focus.

Toward a broader vision of academic resilience

What is Academic Resilience? In its report entitled Against the Odds: Disadvantaged Students Who Succeed in School (2011), the OECD acknowledged “there is no one commonly-used definition of resilience” (p. 22), and further acknowledged its own definition of resilience was limited by the measures used in its survey and the educational outcomes measured by PISA. For the purposes of comparison, the OECD defines disadvantaged youth as those in the lowest one-third of its socioeconomic indicator within each country, based on information about parents’ occupation(s) and along with measures of household possessions. Thus, every country has one-third of its PISA sample defined as being disadvantaged for the purposes of determining levels of resilience. Resilience is then defined as those students in the bottom one-third who perform in the top one-third of its outcome measures. These outcome measures were defined both nationally and internationally, allowing for resilience to be determined within a country and also across countries.

The purpose here is not to critique the OECD as the combined PISA survey and assessment provide a number of different measures beyond performance on the three domains, and the OECD has implemented a multitude of procedures and measures to address fundamental limitations when making such international comparisons. Nevertheless, these limitations have important implications when attempting to define and measure academic resilience. As one critical example, the definition of disadvantaged students as being the lowest one-third of the students within each jurisdiction produces an underlying assumption that relative disadvantage is the same in each jurisdiction. This is clearly not the case as some countries have much less varied distributions in terms of socioeconomic indicators. One need only consider the unequal distribution of parents’ education across countries to observe this problem. Parental education has long been associated with higher achievement and it is a common measure of socio-economic status, but these measures are at best only proxies for some more global measure of a “family’s commitment to education” or “ability to commit to education.”

Measurement researchers have been tasked with an even greater responsibility than much of the science of measurement is ready for, the development of measures that allow for comparisons across international jurisdictions. We have long known that any attempt to measure a construct, whether it be literacy, numeracy, socio-economic status, or academic resilience, provides an incomplete representation of the intended construct. Schnepf (2018) provided a robust summary of the ongoing measurement issues that international organizations such as the OECD must address in order to minimize the inherent measurement issues that reduce the validity of the results from surveys, issues that are exasperated when the measures are used across multiple jurisdictions. Differential sampling errors, construct differences and the impact on the measurement models used, differential sampling, and language translation issues are just a few examples of the ongoing challenges that have yet to be fully resolved. OECD is to be commended for their efforts to resolve these challenges, and they are the first to acknowledge the potential impacts of these measurement challenges.

It must also be remembered that the PISA assessment has a singular focus on one age cohort specifically 15-year-old students. The use of a single age cohort is limited in that it does not allow for the examination and comparisons of different age groups and stages of development. There may be sound reasons to focus on 15-year-old students if one is interested in examining the relationships between schooling and specific content domains. Nevertheless, it is important to remember that educational structures vary greatly among national and international jurisdictions. Depending on the country, a 15-year-old child may still be in schools in which they work almost exclusively with a single teacher, be experiencing multiple classes and teachers for the first time, or have been working with multiple teachers for two or three years. These differences in school structure and the timing of changes to school structures have been linked to cross-jurisdictional differences in school pressure reported by 11-, 13-, and 15-year-old children (Klinger et al., 2015). Further, the proportion of 15-year-old students in schools varies greatly across countries, with less advantaged countries having a large proportion of 15-year-old children not being in school (Schnepf, 2018).

Lastly, COVID-19 has demonstrated globally that unexpected events can have very different impacts on education and children’s mental health issues. It is highly likely that further attempts to measure resilience will be hindered by the not yet understood impacts of the pandemic on our children and education systems. And this will be in hindsight, without the knowledge of other future events that may have isolated or global impacts on education systems and children. Combined, these examples highlight the ongoing challenges for those using national and international survey and assessments results as a tool to shape educational policy and then to measure the subsequent impact of those polices. PISA is a triennial, single age cohort tool, with an unequal focus on three learning domains and self-report survey that is inadequate to capture the nuances of academic resilience and how this is manifested over time and across ages. Educational jurisdictions vary greatly in their approach to schooling and responses to unexpected events, making it even more challenging to examine and compare students’ educational outcomes given the differences in learning contexts associated with place, time, and structure.

And yet, the efforts to use such international surveys and achievement measures to make comparisons and direct policy continue. The desire to support and improve the education of our children around the globe is admirable. On the surface, it is difficult to argue against the concept of academic resilience and its potential value to identify those students who have succeeded “against the odds.” Nevertheless, as our work has shown, the results have led to overly simplistic constructs and what we would consider flawed results. Policymakers need to move away from simplistic conclusions based on such flawed results and comparisons (rankings).

It is also fair to acknowledge that some of the previously noted oversights are not necessarily out of ignorance—rather, some are by design as PISA has been strategically used for political purposes. Interestingly, the Organisation for Economic Cooperation and Development’s (2015) own research has suggested the reforms “needed” for coming close to the best performing countries and the actual reforms implemented have been (very) weak. There is also a significant body of research conducted by academics in different parts of the world that have documented the instrumental use of PISA results by policymakers, sometimes to justify pre-existing reforms. For example, Takayama’s (2008) analysis of the Japanese context suggested that PISA 2003 results were used by the Ministry of Education, Culture, Sports, Science, and Technology (MEXT) to legitimize highly contentious policy measures. More specifically, he noted how the reform of the yutori (low pressure) curriculum and the introduction of national testing were under consideration by MEXT prior to the release of PISA results—a finding that converges with other national contexts that display a heightened level of reactivity to test results (Volante, 2015). More recent analyses across Europe also underscore cases of policy refraction—namely, the tendency of policymakers to focus undue attention on a narrow range of policy implications that align with their political predilections (Volante, 2023; Volante & Klinger, under review). Overall, the literature suggests that PISA results, including those that focus on academic resiliency, can be used quite selectively and in a manner that does not faithfully capture the spirit of evidence-based policy development.

Future research

The pursuit of educational equity, both nationally and internationally, is of critical importance, and our attempts to identify potential mechanisms to ameliorate the disadvantages due to socioeconomic imbalances must continue. The concept of academic resilience has the potential to not only identify those children who are able to “beat the odds” but also determine factors that enable these children to find success. Nevertheless, we must acknowledge we are still in the early stages of exploring these potential contributors to equity. Further to the problematic policy borrowing we have identified, future research should work to better describe the population of students who are facing adverse circumstances, whether these be due to socioeconomic or other factors (e.g., isolation). One viable option is for the OECD to work more closely with the Health Behavior in School Aged Children (HBSC) research team. The HBSC survey is a cross-national survey largely sponsored by the European division of the World Health Organisation. Administered every 4 years, the survey focuses on the health and well-being of young people. The survey includes much broader aspects of health than the PISA survey, although with much less focus on academic achievement (based on students’ self-reports of achievement). It would be possible to ensure samples from both the PISA and HBSC overlap, providing a more complete picture of these students, while also furthering our understanding of “academic resilience.”

At the same time, our work to examine the pitfalls and potentials of national and international policy borrowing must also continue. The COVID-19 pandemic has further differentiated the contexts in which children engage in their education, with substantial differences in the amount of direct face-to-face contact children have with teachers and parents. There is little doubt that economic disparities would have exacerbated these differences within communities, with children having home-learning situations that vary in quality and support. Ultimately, this paper underscores the importance of moving beyond a narrow focus on student achievement to more carefully consider broader health and well-being dimensions that are equally important for future success.