National Gross Domestic Product,Science Interest,and Science Achievement: A Direct Replication and Extension of the Tucker-Drob,Cheung,and Briley (2014) Study

Abstract

We replicated the study by Tucker-Drob, Cheung, and Briley (2014), who found that the association between science interest and science knowledge depended on economic resources at the family, school, and national levels, using data from the 2006 Programme for International Student Assessment (PISA). In more economically prosperous families, schools, and nations, student interest was more strongly correlated with actual knowledge. Here, we investigated whether these results still held despite substantial changes to educational and economic systems over roughly a decade. Using similar data from PISA 2015 (N = 537,170), we found largely consistent results. Students from more economically advantaged homes, schools, and nations exhibited a stronger link between interests and knowledge. However, these moderation effects were substantially reduced, and the main effect of science interest increased by nearly 25%, driven almost entirely by families of low socioeconomic status and nations with low gross domestic product. The interdependence of interests and resources is robust but perhaps weakening with educational progress.

Keywords

science interest science achievement economic resource cross-national expectancy value replication open data open materials preregistered

Demand for science, technology, engineering, and mathematics (STEM) graduates is growing rapidly, but the pipeline of graduates has not increased at a similar rate (Noonan, 2017). Students with less science knowledge may face employment disadvantage (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 2010), and training future STEM leaders is essential to economic growth (Knudsen, Heckman, Cameron, & Shonkoff, 2006). Additionally, knowledge of science is increasingly important for participation in a world shaped by science-based technology. Thus, identifying key predictors of science achievement and aptitude can aid in selecting promising students, promoting science understanding, and crafting interventions. Here, we focused on two factors that Tucker-Drob, Cheung, and Briley (2014) demonstrated play an interdependent role in science achievement: motivational and socioeconomic factors. Science interest (Hidi & Harackiewicz, 2000; Hulleman & Harackiewicz, 2009) and family socioeconomic status (SES; Sirin, 2005) are both positively correlated with achievement, but student interest has been found to have a stronger association with science achievement among individuals from advantaged backgrounds (Tucker-Drob & Briley, 2012; Tucker-Drob & Harden, 2012a, 2012b). Our goal was to test, using high-quality, large-scale (N = 537,170), cross-cultural data, the extent to which this pattern of results is robust to educational, economic, and technological progress.

Tucker-Drob et al. (2014) examined the link between science interest and science knowledge across levels of family, school, and national socioeconomic advantage in the 2006 Programme for International Student Assessment (PISA) data. They found that across 57 countries, students from advantaged backgrounds tended to exhibit a stronger positive association between science interest and science knowledge. No empirical studies have replicated Tucker-Drob et al.’s (2014) cross-national findings on Science Interest × SES interactions in predicting science achievement. Since the data were collected in 2006, substantial changes to the global economy, educational opportunities, and Internet access have occurred (United Nations Department of Economic and Social Affairs, 2015; The World Bank, 2015e). These innovations may have reduced the dependency between access to resources and the association of science interest and science achievement. Alternatively, education may remain sufficiently stratified that Science Interest × SES interactions persist. Given the increased emphasis on direct replications (Simons, 2014), our primary goal in the present research was to closely replicate the above patterns in a similar sample collected in 2015 using the same measures by the same research initiative for which the 2006 data were collected. Our replication data set is ideal insofar as it was collected by a multicohort initiative specifically designed to assess policy impacts and monitor the educational progress of nations across historical time. Moreover, the present research expanded on the original study by incorporating additional countries and testing correlates of change in regression parameters.

Method

The following methodological details and analytic approach were preregistered on the Open Science Framework (OSF) prior to conducting any analyses.¹ The OSF page includes a script to download all relevant data, run all analyses, and create the tables and figures reported in this article.

Participants

The present sample was drawn from PISA 2015, a testing round that included a focus on scientific skills. PISA is an ongoing international project assessing the academic competency of 15-year-olds among the member countries of the Organisation for Economic Co-operation and Development (OECD) and partner countries and economies. PISA has taken place every 3 years since 2000 and focuses on one particular subject for each cycle (reading, mathematics, or science; OECD, 2017). The most recent wave that focused on science occurred in 2015, and the previous PISA wave that focused on science was in 2006 and was used by Tucker-Drob et al. (2014). The 2015 data set contains a total of approximately 540,000 individual student participants selected to represent the population of 15-year-old students from each of 72 countries (representing approximately 29 million students; Gurria, 2016). As a comparison, the 2006 data set contains a total of approximately 400,000 individual student participants selected to represent the population of 15-year-old students from each of 57 countries (for a list of included countries, see Table S1 in the Supplemental Material available online). More information on the recruitment, procedures, assessment methods, and results of the existing data can be found online in the technical reports compiled by the OECD (2016). PISA used a two-stage stratified sampling design based on students nested within schools nested within countries. Within each cluster, 42 students were typically randomly selected. Minor deviations from this occurred because of geographic considerations (e.g., Russia used three-stage stratified sampling because of its landmass) or school-level considerations (e.g., school size). At least 150 schools were stratified and selected to represent the full target population of 15-year-old students in each of the participating countries (or all schools within a country were selected if there were fewer than 150 in total). Within each school, the target cluster size was selected (e.g., 42 students typically). Thus, roughly 5,250 students per country were selected.

The countries sampled in PISA 2006 and PISA 2015 did not match perfectly. PISA 2006 included four countries not sampled in PISA 2015 (Azerbaijan, Kyrgyzstan, Liechtenstein, and Serbia). PISA 2015 included many countries not included in PISA 2006 (for details, see Tables S1–S3 in the Supplemental Material). Thus, the data sets include matching information for 53 countries, with an additional 19 countries only in PISA 2015.

Participants completed the study material either on a computer (93.2%) or with paper and pencil (6.8%), with assessments on science literacy, which lasted about 2 hr. Science literacy refers to “the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen” (OECD, 2016, p. 43). Additionally, participants completed a 30-min background questionnaire. All materials were validated in field trials and scaled using item-response theory, including international item fit and Item × Country interactions. The materials were developed in two source languages (English and French), double-translated by independent translators using both languages for each nation, and verified by independent translators. Strict procedures for localization and quality control were in place to guarantee that accurate, reliable, and comparable information was obtained from each country. Details of the measures and procedures can be found in the technical reports (OECD, 2016).

Measures

Science interest

Students were asked about their views on broad-science topics. Participants rated how much they agreed with statements indexing their interest and enjoyment in learning about broad-science topics. An example item would be “I generally have fun when I am learning [broad-science] topics.” Participants responded to the items on a scale that ranged from 1 (strongly disagree) to 4 (strongly agree). The same items were used in PISA 2006 and PISA 2015.

Family SES and school SES

PISA assessed student reports on family SES with an index of economic, social, and cultural status. School SES was calculated by nesting the PISA index of family SES to individual schools. This approach is identical to that used by Tucker-Drob et al. (2014).

National gross domestic product (GDP)

Per capita GDP in U.S. dollars for each country in 2015 was collected from The World Bank to index national resources (The World Bank, 2015b). GDP was highly stable from 2006 to 2015 (r = .96).

Potential mediators of the Science Interest × GDP effect

Tucker-Drob et al. (2014) included six nation-level variables that could potentially provide a more mechanistic account of the emergence of the Science Interest × GDP effect. We obtained the 2015 or the most recent statistics of the same indices. These measures are the Gini index (measures income inequality, with larger values indicating a greater concentration of income among fewer individuals; The World Bank, 2015c), Democracy Index (measures the extent to which individuals can freely participate in political functions; “Democracy Index,” 2016), expenditures on research and development (The World Bank, 2015d), Social Justice Index (a broad index with higher values representing lower poverty, greater access to education, labor equality, and access to health services and other public services; Schraad-Tischler & Schiller, 2016), access-to-education dimension (measures whether educational opportunities are available to all individuals within a country; Schraad-Tischler & Schiller, 2016), and social-cohesion dimension (measures perceptions of national unity and fair treatment of all; Schraad-Tischler & Schiller, 2016). Additionally, we included three novel measures: expenditures on education, adolescent fertility, and percentage of the population who are Internet users (The World Bank, 2015a, 2015e, 2015f). Educational expenditure and adolescent fertility have shown unique prediction of socioeconomic outcomes compared with GDP (Blackburn & Cipriani, 2002; Gylfason, 2001). In addition, Internet use may be especially beneficial for the academic performance of lower SES children (Jackson et al., 2006; Vigdor, Ladd, & Martinez, 2014). Table S1 displays these indices for each country.

Science achievement

PISA assessed participants in multiple areas of science achievement. The PISA data file provides five sets of plausible values for latent science literacy proficiency, derived using item-response theory. We pooled results using the Mplus (Muthén & Muthén, 2017) multiple-imputation feature to integrate analyses conducted separately for each set of plausible values.

The OECD continuously refines the achievement items across waves of PISA to ensure that translations are appropriate as new countries are added and to reflect changes in the educational system. Therefore, the specific item content differs from that in the work by Tucker-Drob et al. (2014), but all items were intended to measure the same latent construct, science achievement.

PISA 2015 introduced computer-based assessment (CBA) as the main mode of assessment, whereas in prior cycles (2000–2012), PISA used paper-and-pencil-based assessment (PBA) as the main mode of assessment. To address concerns over whether the assessment modality would influence item parameter estimates, PISA 2015 conducted a trial study using a linking design to address potential modality effects. As a result, PISA 2015 used concurrent item calibration to place the PISA 2015 results and the past PISA results on the same scale. The item-response-theory analyses showed that the correlations between the difficulty parameters for PBA and CBA items were high within each domain (r = .92 for science achievement), and the correlation for the discrimination parameter was .94 for science-achievement items. These results imply that the latent construct of science achievement was measured similarly across assessment modalities. The field testing, measurement-invariance testing, and validity of the item content were documented in the PISA technical report (OECD; 2016).

As a robustness check, we estimated our key model on four subsets of participants depending on the assessment modality used in the country: PISA 2006 participants from countries using PBA in both 2006 and 2015, PISA 2006 participants from countries that switched from PBA to CBA, PISA 2015 participants using PBA, and PISA 2015 participants using CBA. Put differently, we tested whether our primary results held regardless of measurement modality. These results are described in the text and in Table S2 of the Supplemental Material. Our primary results are consistent regardless of whether analyses were run using data from PBA and CBA.

Data analysis

We followed the analytic approach of Tucker-Drob et al. (2014) exactly and used the same analytic scripts included as supplemental materials to that publication. We constructed variables in an identical manner (e.g., log-transforming national GDP; using both the U.S. and the pooled standard deviation as a reference), and we built our regression models with a similar progression by including new terms to test the sensitivity of the results.

We conducted analyses on three data sets. First, we reanalyzed data from PISA 2006 using only countries that were also included in PISA 2015. This reanalysis ensured that we made direct comparisons across the data sets to avoid the mismatch in countries obscuring results. Second, we analyzed data from PISA 2015 using only countries that were also included in PISA 2006. We compared the results of these first two sets of analyses as a direct replication. If model parameters differed across these two sets of analyses, then it could not be because of differences in the countries sampled and must be because of some difference in the economic, educational, or political system across time (in addition to random noise). Third, we conducted new analyses based on the full PISA 2015 data set. This extension analysis demonstrates whether the previous findings on matched countries are generalizable to a new set of countries and increases power to detect national-level differences.

We fitted the same series of models as Tucker-Drob et al. (2014). We estimated a series of regression equations on the full data set, taking into account the nesting of students within schools and within countries using the complex-survey option of Mplus (Version 8.0; Muthén & Muthén, 2017). We tested three models. Model 1 regressed science achievement on science interest, family SES, log GDP, Science Interest × Family SES, and Science Interest × Log GDP. Model 2 additionally included school SES and Science Interest × School SES. Model 3 included quadratic terms for science interest and SES. Following Tucker-Drob et al. (2014), we centered all individual-level variables within countries. We used two approaches for standardization: standardization relative to the U.S. standard deviation and standardization relative to cross-nationally pooled standard deviation. We focus on the U.S. standardization here and report the cross-national standardization in Table S5 in the Supplemental Material.

To estimate independent regression coefficients for each country, we regressed science achievement on science interest, family SES, and their interaction in a multigroup model, with each country serving as a group. Then, we extracted these regression coefficients into a metadata set and correlated them with national-level variables, such as national GDP.

Results

We specified three multilevel models to test our hypotheses and reported the effect sizes and 95% confidence intervals (CIs) from both PISA 2006 and PISA 2015. In the initial model, we examined the interactions between family-level SES and science interest in predicting science achievement while controlling for other predictors. In the second model, we included school-level effects, and in the third model, we included quadratic terms to test for nonlinearity. Given the robust design and the well-powered nature of the original study, we primarily focused on effect-size estimates.

Replicable findings

Table 1 displays regression parameters based on three data sets: 53 common countries in PISA 2006, 53 common countries in PISA 2015, and the full 72 countries in PISA 2015. Similar to the results of Tucker-Drob et al. (2014) and our reanalysis using common countries, the results of our first model revealed that science interest (b = 0.187), family SES (b = 0.270), and log GDP (b = 0.406) predicted science achievement in PISA 2015. The interaction effect between science interest and family SES was also found (b = 0.028) as well as between science interest and log GDP (b = 0.063). After school SES was included, the effect of family SES on science achievement was reduced by half (b = 0.114), and school SES remained a moderate predictor of science achievement in PISA 2015 (b = 0.259; Model 2). These results were remarkably similar to those of PISA 2006. We also replicated the small effects of quadratic terms of both family SES and science interest, and we found that the focal family SES and log GDP interactions were robust to the inclusion of the quadratic terms (see Table 1, Model 3). When expanding to the full PISA 2015 data set, we found that results were essentially unchanged, indicating that the current results also generalize to additional countries. One exception was that the main effect of log GDP was reduced in magnitude, possibly because of the inclusion of more heterogeneous countries.

Table 1.

Results of Regression Models Predicting Science Achievement

	Model 1			Model 2			Model 3
	53 Countries	53 Countries	72 Countries	53 Countries	53 Countries	72 Countries	53 Countries	53 Countries	72 Countries
	2006	2015	2015	2006	2015	2015	2006	2015	2015
Effect type and predictor	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)	b (U.S. SD)
Main effect
Family SES	0.278[0.266, 0.290]	0.270[0.258, 0.282]	0.267[0.255, 0.279]	0.113[0.105, 0.121]	0.114[0.104, 0.124]	0.107[0.099, 0.115]	0.109[0.101, 0.117]	0.110[0.100, 0.120]	0.104[0.096, 0.112]
School SES				0.243[0.227, 0.259]	0.259[0.243, 0.275]	0.264[0.244, 0.284]	0.246[0.230, 0.262]	0.262[0.246, 0.278]	0.266[0.248, 0.284]
Log GDP	0.369[0.345, 0.393]	0.406[0.388, 0.424]	0.320[0.304, 0.336]	0.356[0.336, 0.376]	0.397[0.381, 0.413]	0.312[0.296, 0.328]	0.345[0.325, 0.365]	0.388[0.372, 0.404]	0.305[0.289, 0.321]
Science interest	0.170[0.158, 0.182]	0.187[0.179, 0.195]	0.184[0.176, 0.192]	0.184[0.174, 0.194]	0.189[0.181, 0.197]	0.185[0.177, 0.193]	0.190[0.180, 0.200]	0.199[0.191, 0.207]	0.194[0.186, 0.202]
Quadratic effects
Family SES²							−0.035[–0.041, –0.029]	−0.022[–0.028, –0.016]	−0.017[–0.025, –0.009]
Science interest²							0.015[0.009, 0.021]	0.023[0.017, 0.029]	0.016[0.012, 0.020]
Within-country interactions
Science Interest × Family SES	0.040[0.032, 0.048]	0.028[0.020, 0.036]	0.031[0.025, 0.037]	0.010[0.002, 0.018]	0.015[0.007, 0.023]	0.017[0.009, 0.025]	0.012[0.004, 0.020]	0.016[0.008, 0.024]	0.017[0.009, 0.025]
Science Interest × School SES				0.042[0.032, 0.052]	0.012[0.004, 0.020]	0.015[0.007, 0.023]	0.041[0.031, 0.051]	0.011[0.003, 0.019]	0.014[0.006, 0.022]
Person × Nation interaction
Science Interest × Log GDP	0.086[0.072, 0.100]	0.063[0.053, 0.073]	0.059[0.049, 0.069]	0.075[0.063, 0.087]	0.059[0.051, 0.067]	0.056[0.048, 0.064]	0.074[0.062, 0.086]	0.058[0.048, 0.068]	0.055[0.047, 0.063]

Note: Following Tucker-Drob, Cheung, and Briley (2014), we centered all predictors within country and scaled them relative to the U.S. standard deviation; log gross domestic product (GDP) was standardized relative to group level. All models used TYPE = COMPLEX and CLUSTER commands to correct standard errors for nonindependent observations across different levels of analysis (individuals, schools, and country). Values in brackets are 95% confidence intervals. SES = socioeconomic status.

In our multigroup approach, the key result was that countries with higher GDP tended to also display a stronger link between science interest and knowledge was nearly identical across 2006 data (r = .68; see Fig. 1a) and 2015 data (r = .76; see Fig. 1b). Figure 1c plots the interest–knowledge effect size against log GDP for the full sample. The link with log GDP was somewhat smaller (r = .58) in the full data because of some prominent outliers, such as Malta and Lebanon, both of which displayed much larger associations between science interest and science knowledge than their log GDP would imply. Lebanon also displayed the largest Science Interest × Family SES interaction of the new countries (b = 0.106), nearly 5 times larger than the average. All nation-specific effect sizes can be found in Tables S3 and S4 in the Supplemental Material, including those for all countries in PISA 2015.

Fig. 1.

Scatterplots (with best-fitting regression lines) showing the relation between log-transformed per capita gross domestic product (log GDP) and the regression coefficient for the association between science interest and science achievement from (a) 2006, (b) 2015 (53 countries), and (c) 2015 (72 countries). The shaded area represents the 95% confidence interval; the darker shaded region represents the extent of data coverage for log GDP. Data were obtained from Programme for International Student Assessment (PISA) 2006 and PISA 2015. Log GDP is scaled relative to between-country standard deviations. Values on the y-axis are scaled relative to the standard deviation observed in the U.S. subsample. DZA = Algeria; ARG = Argentina; AUS = Australia; AUT = Austria; BEL = Belgium; BRA = Brazil; BGR = Bulgaria; BSJG = Beijing-Shanghai-Jiangsu-Guangdong, China; CAN = Canada; CHL = Chile; COL = Colombia; CRI = Costa Rica; HRV = Croatia; CZE = Czech Republic; DNK = Denmark; DOM = Dominican Republic; EST = Estonia; FIN = Finland; FRA = France; GEO = Georgia; DEU = Germany; GRC = Greece; HKG = Hong Kong, China; HUN = Hungary; ISL = Iceland; IDN = Indonesia; IRL = Ireland; ISR = Israel; ITA = Italy; JPN = Japan; KAZ = Kazakhstan; JOR = Jordan; KOR = Republic of Korea; XKX = Kosovo; LBN = Lebanon; LVA = Latvia; LTU = Lithuania; LUX = Luxembourg; MAC = Macao, China; MYS = Malaysia; MLT = Malta; MEX = Mexico; MDA = Moldova; MNE = Montenegro; NLD = The Netherlands; NZL = New Zealand; NOR = Norway; PER = Peru; POL = Poland; PRT = Portugal; PRI = Puerto Rico; QAT = Qatar; ROU = Romania; RUS = Russian Federation; SGP = Singapore; SVK = Slovak Republic; VNM = Vietnam; SVN = Slovenia; ESP = Spain; SWE = Sweden; CHE = Switzerland; THA = Thailand; TTO = Trinidad and Tobago; ARE = United Arab Emirates; TUN = Tunisia; TUR = Turkey; MKD = Macedonia; GBR = United Kingdom; USA = United States; URY = Uruguay.

Differences across nearly a decade of development

Results from the same 53 countries in 2006 and 2015 showed sizeable decreases in the effect size of the interaction between science interest and economic factors at both intranational and international levels (Science Interest × Family SES—2006: b = 0.040, 2015: b = 0.028; Science Interest × GDP—2006: b = 0.086, 2015: b = 0.063). Holding log GDP constant, analyses of the 2006 results indicated that the association between science interest and science achievement was .090 among children from low-SES families (i.e., 2 SD below the mean) and .250 among children from high-SES families (i.e., 2 SD above the mean), but the 2015 results indicated that children from low-SES families displayed a larger effect size (0.131) compared with a largely unchanged effect size (0.243) for high-SES children. The results focusing on log GDP were similar. The gap between the effect size for students in prosperous nations compared with developing nations shrunk by .092 correlation units over the decade, with nearly the entire shift due to an increase in the strength of the association among developing countries. For prosperous nations, the association was estimated to be .342 in 2006 and still .313 in 2015. Additionally, we were curious whether changes in GDP across time could explain these deviations. However, we found that replacing 2015 GDP with 2006 GDP did not produce different results, consistent with very high rank-order stability of GDP. These results point toward changes in the other variables playing a larger role in the differences.

The Science Interest × School SES interaction also substantially decreased compared with Tucker-Drob et al.’s (2014) findings (2006: b = 0.042; 2015: b = 0.012). The 95% CI for these estimates did not overlap. When we included the Science Interest × School SES interaction in the model (Model 2), the Science Interest × Family SES interaction decreased in magnitude by half compared with the interaction in Model 1 in the 2015 data, but this effect decreased by 70% in the 2006 data. Thus, school SES was a less plausible mediator of the Science Interest × Family SES interaction in the 2015 sample.

Differences were also evident in our multigroup model. Although the correlation between log GDP and the association between science interest and knowledge was essentially equal across the data sets, the pattern was strikingly different. In 2006, the association between interest and knowledge ranged from below 0 to .3, but in 2015, countries at the lower end of the log-GDP distribution were shifted up, shrinking the distribution to between .1 and .3. Interestingly, the higher end of the distribution was constant. This effect is more apparent in Figure 2a, which plots the country-specific 2006 interest effect size against the country-specific 2015 interest effect size. In the upper-right quadrant of the scatterplot, a nearly perfect linear relation is distributed symmetrically around the 45° reference line. However, the lower-left quadrant displays many countries with substantially larger effect sizes in 2015 compared with 2006. Several countries with the largest discrepancy are in South America (e.g., Colombia, Argentina, Brazil, and Uruguay) and Eastern Europe (e.g., Montenegro, Romania, Russia, Bulgaria, Slovenia, and Poland).

Fig. 2.

Scatterplots showing the association between coefficients obtained in Programme for International Student Assessment (PISA) 2006 and PISA 2015, separately for (a) science interest, (b) family socioeconomic status (SES), and (c) the Science Interest × Family SES interaction. A 45° reference line has been added to each graph. Data points above the line represent coefficients that were larger in 2015 than in 2006. Data points below the line represent coefficients that were larger in 2006 compared with 2015. Because the range of Science Interest × SES effect sizes was small, (c) includes an inset that presents a magnified representation of the scatterplot. The magnified section is surrounded by a box. Coefficients are scaled relative to the standard deviation observed in the U.S. subsample. See the Figure 1 caption for country abbreviations.

In contrast, Figure 2b plots the country-specific 2006 family-SES effect sizes against the 2015 estimates, and the association was strong (r = .83) and linear. Finally, Figure 2c displays the nearly null association between the Science Interest × Family SES interaction term across the two data sets. The lack of association may be due to the relatively small range of effect sizes—between 0 and 0.1 in both data sets—or the relative statistical imprecision of estimating interaction effects compared with main effects.

National-level predictors of science-interest effect size beyond GDP

Because of the large correlation between log GDP and other national-level economic or political indicators, Tucker-Drob et al. (2014) used commonality analysis to determine whether more proximate national-level variables (e.g., educational policy) might provide insight into the effect. Similar to those of Tucker-Drob et al. (2014), our results showed that log GDP was the strongest correlate of the country-specific association between science interest and science achievement. Beyond log GDP, the largest unique effects were observed for access to education (31.1% of total R²), adolescent fertility (27.5% of total R²), and the Gini index (15.1% of total R²), as shown in Table S6 in the Supplemental Material.

National-level predictors of change in science-interest effect size

Table S7 in the Supplemental Material reports a similar set of commonality analyses with change in the interest effect size from 2006 to 2015 as the dependent variable. These results indicate what sorts of conditions facilitated or hindered an increase in the association between interest and knowledge across time. As found with the single-time-point estimates, changes in science-interest effect sizes were primarily associated with log GDP (r = −.452; see Fig. S2 in the Supplemental Material), and the other national-level predictors provided relatively little incremental prediction. The largest unique effects were for the Democracy Index (25.0% of total R²), adolescent fertility (19.8% of total R²), and access to education (12.6% of total R²).

We were also interested in examining each predictor and changes in predictors in isolation. Table S8 in the Supplemental Material reports correlations between national-level predictors and the science-interest effect size in 2006 and 2015, the change in interest effect size across these time points, and the correlated change across both the national-level predictors and the interest effect size. Correlations with 2006 and 2015 science-interest effect sizes were largely similar. However, results show that differences in magnitudes of the main effect of science interest across PISA 2006 and PISA 2015 were negatively associated with log GDP, Democracy Index, research and development expenditures, social cohesion, and access to the Internet (all rs < −.3). Put differently, these results indicate that the association between science interest and achievement has increased more in less prosperous countries than in rich countries over the decade, confirming our qualitative assessment of Figure 2a. High adolescent fertility was also associated with a positive change in the science-interest effect size (r = .439). Countries with relatively high adolescent fertility in 2015 tended to increase in the science-interest effect size to a greater extent from 2006 to 2015.

Finally, countries in which the connection strengthened between science interest and science knowledge tended to increase more in indicators of resource accessibility, such as access to the Internet (r = .489), access to education (r = .363), and education expenditure (r = .372; see the last column of Table S8, which presents change-change correlations between the national-level predictor and the science-interest effect size). Interestingly, change in log GDP was not correlated with change in the interest effect size (r = .068), perhaps because of the highly stable nature of GDP. These results are consistent with our hypothesis that increasing the availability of educational resources or removing structural barriers to learning would more easily allow students to transform interest into knowledge.

Discussion

Tucker-Drob et al. (2014) found that students with higher science interest had higher science knowledge. The magnitude of this association varied across families and nations, with more economically prosperous families and nations showing a stronger association. We closely replicated these results with new data. Our results indicate that Tucker-Drob et al.’s (2014) key findings are robust. In the present sample, science interest interacted with both family SES and national GDP in predicting science achievement. Students with a greater drive to learn were more likely to possess science knowledge when economic resources were available. This result could be attributable to the differential distribution of educational conditions (e.g., teacher or school quality), material conditions (e.g., accessibility of scientific textbooks in the home), or psychological conditions (e.g., stress caused by food insecurity) that enable learning. Another possibility is that economic resources could allow students to have better informed interests via increased exposure to science content. When student interest is accurately calibrated to the student’s true proclivities and aptitude (rather than vague or uninformed preferences), interest might more strongly predict achievement.

In direction and statistical significance, our results were highly similar to those of Tucker-Drob et al. (2014). However, differences in effect-size estimates are of strong relevance, particularly given the large sample sizes of both the original study and current study (Asendorpf et al., 2013). Two main differences were found across a decade of development.

First, effect sizes for the interaction between national GDP and family SES decreased. Thus, the association between science interest and science knowledge continues to depend on economic factors but to a lesser extent than previously documented. It is possible that societies now provide more equitable access than previously observed through policy changes or technological improvements. If interested, low-income students have more opportunities to acquire knowledge using online platforms (Jackson et al., 2006), such as Khan Academy (Thompson, 2011). This interpretation is consistent with our finding that countries in which Internet and educational accessibility increased more also tended to show greater increases in the association between science interest and science knowledge. Our results indicate that an important difference over the decade is that relatively less-well-off students now evince some links between science interest and knowledge, whereas this association was entirely absent a decade ago. Among wealthier students, the magnitude of the association between interest and knowledge was largely unchanged.

Second, we found low consistency in the magnitude of the Science Interest × Family SES interaction between 2006 and 2015 across nations (see Fig. 2c). There are likely multiple explanations for this finding. Interaction effects are difficult to replicate because they are almost always estimated with less precision and typically are smaller in magnitude compared with main-effect estimates in similar data (for a recent special issue on replication with reasonable success for main effects but poor success for interactions, see Donnellan & Lucas, 2018). When we analyzed the 2015 data across all nations, we obtained a highly statistically significant Science Interest × SES effect size (p = 1.05 × 10⁻¹³). If our effect sizes are typical of common interactions in psychology, even for studies conducted in the same locations, using the same materials, and with thousands of participants, researchers should expect relatively little consistency in effect-size estimates. The more optimistic framing of our results, on the other hand, is that the average absolute deviation from the 2006 interaction effect-size estimate in 2015 was only 0.022 (a deviation that represents approximately one third of the total range of interaction effect sizes in 2015).

Limitations and future directions

The current study highlights the use of existing data to examine and replicate questions pertaining to real-world outcomes. Notably, because the original study used a massive sample size, we expected robust findings. The current study replicated most of the original findings, yet the magnitude of some interactions decreased, suggesting that even when the original study and the replication study use large, high-quality samples, findings can differ across studies. This result points toward true moderators, perhaps academic or economic policy changes across the decade over which the two sets of data were collected.

This study contained the same limitations as the original. As discussed by Tucker-Drob et al. (2014), this was a cross-sectional study using correlational data, so the direction of causal effect could not be determined. The cross-sectional nature of the design raises the possibility that some of the observed differences might reflect cohort effects. In this report, we have demonstrated the generalizability of the effect across many countries and also across two cohorts. Another limitation can be seen in the use of secondary data to develop the national-level predictors. These variables are only indirect markers for between-country socioeconomic differences. More proximal indicators might show even stronger effects. Caution should be taken when interpreting the school-SES results as this predictor of student-level outcomes may operate by indexing cumulative effects that operate via previous achievement (Marks, 2015).

Following recent recommendations to include explicit statements describing the generalizability of research findings (Simons, Shoda, & Lindsay, 2017), we expect that our results would generalize to 15-year-olds attending public school across a wide range of nations. This age was selected by PISA because 15-year-old students are generally close to the end of their compulsory education, which is also a time when career trajectories begin to canalize. Our data set did not include many low-income nations, but we certainly captured economic diversity across nations and, importantly, within nations. Given the broad range of settings and sampling frames, we would expect these results to generalize to most formalized academic settings. Much of the empirical and theoretical work that motivated this study and that of Tucker-Drob et al. (2014) spans early childhood to the college years (see Wigfield & Eccles, 2002). However, given the narrow age range of the participants in the current study, it is not clear how generalizable these findings are to different periods of development.

Future work should evaluate the generalizability of the results across academic domains. It is unclear whether the current findings are specific to the science domain, extend to other academic domains, or are attributable to domain-general academic interest. That said, our previous analyses of separate data from more than 375,000 American high school students found evidence for domain-specific Science Interest × SES interactions (e.g., music, sports, literature, and biology; Tucker-Drob & Briley, 2012).

Conclusion

In this direct replication and extension of the study by Tucker-Drob et al. (2014), we found similar results. There was a stronger association between science interest and science knowledge in students from economically advantaged homes and nations. Despite replicating these results, we found differences in parameter magnitude. Student interest was a stronger independent predictor of achievement, and the dependence on economic resources was weaker primarily because of shifts among lower SES families and lower GDP countries. Future work is needed to understand the mechanisms that produce this dependence. One promising avenue that we see is in exploring what allowed countries that increased in access to the Internet and education to see increases in the link between science interest and achievement.

Supplemental Material

Briley_OpenPracticesDisclosure_rev – Supplemental material for National Gross Domestic Product, Science Interest, and Science Achievement: A Direct Replication and Extension of the Tucker-Drob, Cheung, and Briley (2014) Study

Supplemental material, Briley_OpenPracticesDisclosure_rev for National Gross Domestic Product, Science Interest, and Science Achievement: A Direct Replication and Extension of the Tucker-Drob, Cheung, and Briley (2014) Study by Anqing Zheng, Elliot M. Tucker-Drob and Daniel A. Briley in Psychological Science

Supplemental Material

Zheng_Supplemental_Material – Supplemental material for National Gross Domestic Product, Science Interest, and Science Achievement: A Direct Replication and Extension of the Tucker-Drob, Cheung, and Briley (2014) Study

Supplemental material, Zheng_Supplemental_Material for National Gross Domestic Product, Science Interest, and Science Achievement: A Direct Replication and Extension of the Tucker-Drob, Cheung, and Briley (2014) Study by Anqing Zheng, Elliot M. Tucker-Drob and Daniel A. Briley in Psychological Science

Footnotes

Action Editor

Steven W. Gangestad served as action editor for this article.

Author Contributions

D. A. Briley developed the study concept. A. Zheng and D. A. Briley conducted the analyses and drafted the manuscript. E. M. Tucker-Drob provided critical input on the analytic plan and manuscript. All the authors approved the final manuscript for submission.

Declaration of Conflicting Interests

The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article.

Supplemental Material

Additional supporting information can be found at

Open Practices

Data for this study were obtained from the Organisation for Economic Co-operation and Development’s Programme for International Student Assessment and can be accessed at http://www.oecd.org/pisa/data/2015database/. All data and scripts necessary to replicate the analyses have been made publicly available via the Open Science Framework and can be accessed at osf.io/t7jjk. This study was preregistered at https://osf.io/frwtb/register/5730e99a9ad5a102c5745a8a. The preregistered analysis plan was altered slightly after the analysis began by removing two country-level variables and adding one (see Note 1). The complete Open Practices Disclosure for this article can be found at https://journals-sagepub-com.web.bisu.edu.cn/doi/suppl/10.1177/0956797619835768. This article has received the badges for Open Data, Open Materials, and Preregistration. More information about the Open Practices badges can be found at .

Notes

References

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J.

Fiedler

Perugini

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.

Blackburn

Cipriani

G. P.

(2002). A model of longevity, fertility and growth. Journal of Economic Dynamics & Control, 26, 187–204.

Democracy Index 2015: Democracy in an age of anxiety. (2016). The Economist. Retrieved from https://www.eiu.com/public/topical_report.aspx?campaignid=DemocracyIndex2015

Donnellan

M. B.

Lucas

R. E.

(2018). Introduction to the special issue—A replication project in personality psychology. Journal of Research in Personality, 72, 1–4.

Gurria

(2016). PISA 2015 results in focus. PISA in Focus (No. 67). Retrieved from https://www.oecd-ilibrary.org/docserver/aa9237e6-en.pdf?expires=1552319447&id=id&accname=guest&checksum=745810FED54A4B2912A7BA07A43E9BBB

Gylfason

(2001). Natural resources, education, and economic development. European Economic Review, 45, 847–859.

Hidi

Harackiewicz

J. M.

(2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70, 151–179.

Hulleman

C. S.

Harackiewicz

J. M.

(2009). Promoting interest and performance in high school science classes. Science, 326, 1410–1412.

Jackson

L. A.

von Eye

Biocca

F. A.

Barbatsis

Zhao

Fitzgerald

H. E.

(2006). Does home Internet use influence the academic performance of low-income children? Developmental Psychology, 42, 429–435.

10.

Knudsen

E. I.

Heckman

J. J.

Cameron

J. L.

Shonkoff

J. P.

(2006). Economic, neurobiological, and behavioral perspectives on building America’s future workforce. Proceedings of the National Academy of Sciences, USA, 103, 10155–10162.

11.

Marks

G. N.

(2015). Are school-SES effects statistical artefacts? Evidence from longitudinal population data. Oxford Review of Education, 41, 122–144.

12.

Muthén

L. K.

Muthén

B. O.

(2017). Mplus user’s guide (8th ed.). Los Angeles, CA: Author.

13.

National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2010). Rising above the gathering storm, revisited: Rapidly approaching Category 5. Washington, DC: National Academies Press. doi:10.17226/12999.

14.

Noonan

(2017). STEM jobs: 2017 update. Washington, DC: U.S. Department of Commerce Economics and Statistics Administration.

15.

Organisation for Economic Co-operation and Development. (2016). PISA 2015 technical report. Retrieved from http://www.oecd.org/pisa/data/2015-technical-report

16.

Organisation for Economic Co-operation and Development. (2017). PISA 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving (Rev. ed; pp. 17–46). Paris, France: OECD Publishing.

17.

Schraad-Tischler

Schiller

(2016). Social justice in the EU—Index report 2016: Social inclusion monitor Europe. Gütersloh, Germany: Bertelsmann Stiftung.

18.

Simons

D. J.

(2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80.

19.

Simons

D. J.

Shoda

Lindsay

D. S.

(2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12, 1123–1128.

20.

Sirin

S. R.

(2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75, 417–453.

21.

Thompson

(2011, July 15). How Khan Academy is changing the rules of education. Wired. Retrieved from https://www.wired.com/2011/07/ff_khan/

22.

Tucker-Drob

E. M.

Briley

D. A.

(2012). Socioeconomic status modifies interest-knowledge associations among adolescents. Personality and Individual Differences, 53, 9–15.

23.

Tucker-Drob

E. M.

Cheung

A. K.

Briley

D. A.

(2014). Gross domestic product, science interest, and science achievement: A Person × Nation interaction. Psychological Science, 25, 2047–2057.

24.

Tucker-Drob

E. M.

Harden

K. P.

(2012a). Intellectual interest mediates Gene × Socioeconomic Status interaction on adolescent academic achievement. Child Development, 83, 743–757.

25.

Tucker-Drob

E. M.

Harden

K. P.

(2012b). Learning motivation mediates gene-by-socioeconomic status interaction on mathematics achievement in early childhood. Learning and Individual Differences, 22, 37–45.

26.

United Nations Department of Economic and Social Affairs. (2015). World economic situation and prospects 2015. New York, NY: Author.

27.

Vigdor

J. L.

Ladd

H. F.

Martinez

(2014). Scaling the digital divide: Home computer technology and student achievement. Economic Inquiry, 52, 1103–1119.

28.

Wigfield

Eccles

J. S.

(2002). The development of competence beliefs, expectancies for success, and achievement values from childhood through adolescence. In Wigfield

Eccles

J. S.

, Development of achievement motivation (pp. 91–120). San Diego, CA: Academic Press.

29.

The World Bank. (2015a). Adolescent fertility rate (births per 1,000 women ages 15-19). Retrieved from https://data.worldbank.org/indicator/sp.ado.tfrt

30.

The World Bank. (2015b). GDP per capita (current US$). Retrieved from https://data.worldbank.org/indicator/NY.GDP.PCAP.CD?page=1

31.

The World Bank. (2015c). GINI index. Retrieved from https://data.worldbank.org/indicator/SI.POV.GINI?page=1

32.

The World Bank. (2015d). Government expenditure on education (% of GDP). Retrieved from https://data.worldbank.org/indicator/SE.XPD.TOTL.GD.ZS?page=1

33.

The World Bank. (2015e). Individuals using the Internet (% of population). Retrieved from https://data.worldbank.org/indicator/IT.NET.USER.ZS

34.

The World Bank. (2015f). Research and development expenditure (% of GDP). Retrieved from https://data.worldbank.org/indicator/GB.XPD.RSDV.GD.ZS?page=1

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.40 MB

0.00 MB

0.58 MB