Reward Functioning in General and Specific Psychopathology in Children and Adults

Abstract

Objective:

Problems with reward processing have been implicated in multiple psychiatric disorders, but psychiatric comorbidities are common and their specificity to individual psychopathologies is unknown. Here, we evaluate the association between reward functioning and general or specific psychopathologies.

Method:

1,213 adults and their1,531 children (ages 6–12) completed various measures of the Positive Valence System domain from the Research Domain Criteria (RDoC). Psychopathology was assessed using the Child Behavior Checklist for children and the Adult Self Report for parents.

Results:

One general factor identified via principal factors factor analysis explained most variance in psychopathology in both groups. Measures of reward were associated with the general factor and most specific psychopathologies. Certain reward constructs were associated solely with specific psychopathologies but not general psychopathology. However, some prior associations between reward and psychopathology did not hold following removal of comorbidity.

Conclusion:

Reward dysfunction is significantly associated with both general and specific psychopathologies.

Keywords

RDoC reward comorbid psychopathology CBCL

Introduction

Although psychiatric disorders are delineated as discrete categories in the Diagnostic and Statistical Manual of Mental Disorders (DSM), comorbidity among disorders is common, with as many as 45% of patients fulfilling the diagnostic criteria for multiple disorders in 1 year (Caron & Rutter, 1991). In response to this challenge, among others, the US National Institute of Mental Health launched the Research Domain Criteria (RDoC) initiative (Insel et al., 2010). It provides an alternative framework for research of psychiatric diseases through the use of broad functional dimensions of human behavior. The RDoC paradigm postulates that aberrations in research-validated, normal neurobehavioral functions or constructs are responsible for disorder development. In turn, these “cross-disorder” constructs can be evaluated by both neurobiological and behavioral measures, and are grouped into higher domains that encapsulate similar processes (Cuthbert, 2014). One such domain is the Positive Valence Systems, which describes processes that direct “responses to positive motivational situations,”; constructs that fall under this umbrella include Reward Valuation, Reward Responsiveness, and Reward Learning (NIMH, 2016).

Other analytic approaches have also emerged to further investigate the issue of psychiatric comorbidity. One such method is factor analysis; here, comorbid disorders are grouped into higher-order factors such as “internalizing” or “externalizing” disorders. However, despite this classification, significant covariation continues to exist between these dimensions, with both clinical and genetic studies implicating a single factor, the P-factor, that reflects the presence of a variety of symptoms that cut across different psychopathologies (Caspi et al., 2014; Lahey et al., 2014; Selzam et al., 2018; Smoller et al., 2019; Waldman et al., 2016). Regarded as analogous to the g of general intelligence by some, the presence of the P-factor has motivated studies of nonspecific and specific risk factors of psychopathology (Jensen, 1993; Spearman, 1904).

Altered reward functioning has been previously linked with different individual psychiatric disorders. A steeper gradient in delayed discounting has been identified in patients with attention-deficit/hyperactivity disorder (ADHD), addictive disorders, major depressive disorder (MDD), and other conditions (Amlung et al., 2017, 2019; de Castro Paiva et al., 2019). Substance abuse disorders (SUDs) and MDD have been associated with anhedonia and increased risky decision-making, as measured by the Iowa Gambling Task (IGT) (Kovács et al., 2017; Rizvi et al., 2018; Siqueira et al., 2018). Additionally, increased impulsive decision-making is found in children with ADHD, though results have been mixed for adults with ADHD (Groen et al., 2013). In addition to a heightened incidence of anhedonia, risky behavior, and reduced effort-based decision making, fMRI studies have shown aberrant reward processing in MDD patients (Keren et al., 2018; Pulcu et al., 2014). Finally, altered neural reactivity to reward has been observed in individuals exhibiting antisocial behavior (Murray et al., 2017, 2018).

The relationship between reward and general psychopathology is not well studied. Understanding the degree to which different reward constructs are independently associated with P-factor and specific psychopathologies will enhance our understanding of the link between reward and psychiatric illness. We hypothesized that both the P-factor and specific disorder problem severity would be independently associated with reward functioning but did not have a directional hypothesis about their relative contributions to this association. In our analysis, we extracted P via factor analysis and derived psychopathology specific scores; we then tested the association between reward measures and these measures of general and specific psychopathology.

Methods

Study Cohort: Child probands in the age range of 6 to 12 years were ascertained; family members of the child participants served as the adult cohort, with no intentional weighting of the non-pediatric sample. Participants were recruited from a variety of sources, including on-site recruiting at summer festivals at community programs and sporting centers, a Facebook page, Craig’s List, a posting on the SUNY Upstate Medical University’s clinical trials website, the distribution of pamphlets to general pediatric offices, and word of mouth. Recruitment also occurred via StudyKIK, a website where patients can identify and sign up for clinical trials; patients would find the site via social community pages and Google. To capture subjects with higher psychopathology, brochures and flyers were shared in Psychological Services Offices and Community Mental Health Organizations, including the Child and Adolescent Psychiatry Clinic at SUNY Upstate Medical University. Presentations were also made to an outpatient mental health clinic, organizations providing behavioral and mental health support to adolescents and young adults, as well as school based substance abuse prevention programs.

After rapport was established with the proband and their family, the study was described to them, and they were presented with the voluntary opportunity to participate in the study. Adults were only included after they provided signed informed consent for experimentation with human subjects following explanation of the study, while minor subjects were required to assent to the study with at least one parent providing informed consent prior to participation.

Participants: Inclusion forms were completed at each visit by adults to ensure subject eligibility; adults and children who had sensorimotor disabilities, diagnosed neurological disease, a history of head trauma with a documented loss of consciousness exceeding 10 min, an uncontrolled medical condition, or lack of comprehension of the English language, were excluded from the study to avoid possible confounding effects. In addition, adopted children as well as adults who could not independently complete the tasks of the study, and pregnant women or those who gave birth up to 6 months prior to the study visit, were not retained as participants. Potential participants who had an estimated intelligence quotient (IQ) below 80, as computed via the vocabulary and abstraction subtests of the Shipley-2, a well validated assessment of crystallized and fluid cognitive ability, appropriate for individuals with ages 7 to 89, were excluded from the study (Kaya et al., 2012; Shipley, 1940) . Parents older than 59 were excluded to minimize effects of possible cognitive decline. In addition to inclusion forms, parents also completed mental health questionnaires for themselves and their children to determine mental health status. The investigation was carried out in accordance with the latest version of the Declaration of Helsinki and the study design reviewed and approved by the SUNY Upstate institutional review board.

Population: 1513 children, with an average age of 9 years (SD = 2.1) and1,232 parents between the ages of 23 and 59 years (mean = 37 years, SD = 6.8) made up the study population. The study population was enriched for psychopathology relative to the general population, with 41% of the child probands and 54% of the adult participants reporting a psychiatric history, as defined by having experiences of previously seeking mental healthcare for emotional or behavioral issues. The sample had nearly equal numbers of male and female juvenile participants (51% to 49%, respectively), albeit females were overrepresented in the parental population, (70% female, 30% male). Fifty-eight percent of children and 68% of parents identified as White, 23% of children and 22% of parents as Black, and 19% of children and 10% of adults as “other” or multiple races. In addition, approximately 12% of children and 7% of parents identified as Hispanic. Altogether, the dataset consisted of 950 different families with an average size of 2.94 individuals. Variations of this dataset have been used in prior studies (Albert et al., 2020; Nguyen et al., 2019).

Measures: The study visit lasted for approximately 3 hr, with participants completing several tasks and behavioral assessments. We opted to use broadband measures of psychopathology due to their ability to cover a wide range of psychopathologies efficiently; this approach has been recommended when transdiagnostic frameworks are being studied (Stanton et al., 2020).

Adult Self Report (ASR): Psychopathology in adult participants was assessed via the ASR, a self-reported, 126-item questionnaire, employed for ages 18 to 59 (Achenbach & Rescorla, 2003). It is a broadly utilized instrument that evaluates psychopathology, substance use, and adaptive functioning (Achenbach & Rescorla, 2003). T-scores for symptoms of six DSM disorders (depressive disorders, anxiety disorders, somatic problems, ADHD, avoidant personality, and antisocial personality) were computed via the ASR (T. M. Achenbach et al., 2005). It also provides subscales for symptoms of substance abuse (tobacco, alcohol, recreational drugs), as well as a composite total for substance abuse (Achenbach & Rescorla, 2003). In addition, T-score scales are available to describe symptoms of obsessive-compulsive problems (OCP), sluggish cognitive tempo (SCT), as well as stress problems and total problems and measures of internalizing and externalizing behavior (Achenbach & Rescorla, 2003). In this study, T-scores from individual substance abuse subscales, all DSM scales, and OCP and SCT were approximated as measures of problems linked to specific psychopathologies (Achenbach & Rescorla, 2003).

Child Behavior Checklist (CBCL): Psychopathology in children participating in the study was assessed via the CBCL, a 113-item, parent-reported questionnaire (Achenbach & Edelbrock, 1991; Ivanova et al., 2007) It assesses behavioral and emotional problems in children from ages 6 to 18 (Ivanova et al., 2007). Six DSM scales are also available for the CBCL (affective problems, anxiety problems, somatic problems, ADHD, oppositional defiant problems and conduct problems), as are scales for SCT and obsessive compulsive problems (OCP) (Nakamura et al., 2009). T-scores from these scales were employed in the study.

Reward Measures: Children completed the Experienced Pleasure Scale for Children (EPSC), Iowa Gambling Task (IGT), Effort Expenditure for Rewards Task (EEfRT), Delayed Discounting Task (DDT) and Probability Discounting Tasks (PDT) (Bull et al., 2015; Cauffman et al., 2010; Garon et al., 2006; Kazdin, 1989; Nguyen et al., 2019; Richards et al., 1999; Scheres et al., 2010; Treadway et al., 2009). Adults performed the IGT, EEfRT, DDT, PDT, Temporal Experience of Pleasure (TEPS), and Behavioral Activation System (BAS) tasks (Carver & White, 1994; Gard et al., 2006). Summary measures for the aforementioned tasks were included in the study; for the IGT, EEfRT, and BAS, two analytic variables were included, the individual IGT net earnings (IGT-NE) and total latency (IGTL) the Effort Expenditure for Rewards Choice total, (EERCT) and the Effort Expenditure for Rewards Beta coefficient (EERCB), and the BAS drive and reward responsiveness scale values (BASD, BASR) (Bull et al., 2015; Carver & White, 1994; Nguyen et al., 2019). See Supplemental Materials for details.

Measures were selected based on past reliability and their ability to evaluate similar RDoC PVS constructs and subconstructs in both children and adults. In some cases, the same instrument could not be used for both age groups, so two or more were used. In children, Reward Valuation, Effort Valuation, Reward Expectancy/Prediction Error and Initial Responsiveness to Reward Attainment were evaluated by the DDT and PDT, the EEfRT, the EPSC and IGT-NE, and the EPSC and IGTL, respectively. In adults, Reward Valuation, Effort Valuation, Reward Expectancy/Prediction Error and Initial Responsiveness to Reward Attainment were evaluated by the DDT, PDT, and BASR, the BASD, the TEPS and IGT-NE, and the TEPS and IGT-NL, respectively. Quality control was ensured by use of standardized instructions for each measure or assessment as well as completion of study chart and visit checklists to ensure that all procedures and necessary assessments were performed.

Statistical Analyses: Statistical analyses were conducted with Stata 15.1. We adjusted for age, sex, race, and ethnicity (via dummy variable coding) through ordinary least squares regression. To resolve missing income data for approximately 25% of adults, multiple imputation with 10 iterations was employed as previously described; imputation was not performed for any other variables (Nguyen et al., 2019). To derive the P-factor, a bifactor confirmatory factor analysis, as used in previous studies, was attempted (Caspi et al., 2014). In addition, principal factors factor analysis (PFFA) was conducted using T-scores for the psychopathologies selected for each age group. To evaluate the role of item overlap between the DSM scales and SCT and OCP (four items per age group), PFFA to identify the P-factor was also done exclusively with DSM scales. After the loading matrix was computed, factors with eigenvalues greater than 1 were identified, and then the matrix underwent an orthogonal varimax rotation. The factor with the highest eigenvalue, that is, Factor 1, was designated the P-factor. Based on this, scores for Factor 1 were assigned to each individual as their P-factor values. In addition, the loadings for each individual disorder on Factor 1 were obtained; these values were considered representative of how much variance within each disorder could be explained by the general factor of psychopathology.

To obtain T-score values representative of disorder-specific pathology, T-scores for individual disorders were linearly regressed against P-factor scores. The residuals computed were regarded as disorder problem specific T-scores. Factor analysis was conducted on the reward variables to evaluate potential overlap between measures. To identify which subset of reward variables were best suited for predicting the P-factor and disorder problem specific T-scores, the Furnival-Wilson leaps-and-bounds algorithm using logarithmic likelihoods was applied through the Stata function “GVSelect.” The variables in the models that yielded low Bayesian and Akaike information criterions were used in linear regression.

The relationships between the reward measures, the P-factor, and disorder-specific T-scores were evaluated by linear regression analysis. The F-test of the overall significance was used to assess whether the null hypothesis could be rejected and that the model with the proposed variables provided superior fit to a constant-only model. In order to account for the increased risk of type-I error due to multiple tests, the Benjamini-Hochberg method was used to adjust calculated p-values (Chen et al., 2017). Nine tests were conducted for the child data and 12 for the adult data. To determine statistical significance, we used a 5% False Discovery Rate. Two-tailed test p-values were computed for each regression coefficient and similarly compared to the alpha.

To evaluate the univariate relationship between reward measures and disorders, Pearson’s correlation coefficients were computed between reward measures and disorder problem severity. Correction for multiple testing was addressed as described for the regression analysis. 17 and 24 tests were conducted for child and adult data, respectively.

Results

Factor Analyses of Psychopathology: The Bifactor Confirmatory Factor Analysis failed to converge; thus, PFFA was alternatively used. PFFA for both child and adult disorders yielded only one factor, that is, the P-factor, with an eigenvalue greater than 1. In children, P explained 54% of variance, and in adults, over 95%. Overall, anxiety problems loaded the highest on the P-factor in children, with externalizing disorders such as oppositional defiant disorder and conduct disorder loading the lowest (Figure 1). In adults, all non-substance use-related psychopathologies loaded substantially on P, with substance pathologies showing very low loadings (Figure 2). PFFA performed exclusively using data from subjects recruited from non-enriched sites yielded similar results to inclusive analyses for both age groups; P explained 54% of variance in children and 94% in adults.

Figure 1.

Loadings for specific disorders on the common factor, “P,” in children.

Figure 2.

Loadings for specific disorders on the common factor, “P,” in Adults.

PFFA on datasets with OCP and SCT removed similarly yielded one general factor, with it accounting for ~67% of variance in children and ~99% in adults. In both age groups, loadings on Factor 1 were increased; however, the overall patterns were the same.

Reward Measures and Psychopathology in Children: PFFA of reward indicators revealed a seven-factor structure, with each measure loading substantially on only one factor; correspondingly, all measures were retained. Five of the seven measures of reward significantly correlated with the P-factor and disorder-specific psychopathology scores: EERCT, PDT, DDT, IGT-NE, and EERCB (see Table 1). Table 1 shows that nearly all psychopathologies, except for OCP and Somatic problems, were significantly associated with reward measures; however, most disorder problems were linked to only a few reward measures. Table 1 shows the correlations between reward measures and psychopathology; empty table entries indicate that the correlation was not significant. One reward scale (DDT) was associated with both general psychopathology and specific psychopathologies, while the rest were uniquely associated with disorder problems (see Table 1).

Table 1.

Correlation of Reward Functioning With General and Specific Psychopathologies in Children.

Dependent variable	Independent variables
Dependent variable	Pleasure scale	Energy expenditure for reward choice total	Iowa gambling total net earnings	Probability discounting task	Delayed discounting task	Energy expenditure for reward choice beta coefficient	Iowa gambling total latency	Model F-statistic	Model p-value
P-factor					0.061 (0.028)			5.62	.023
ADHD		−0.074 (0.0097)						8.33	.0060
Depression		−0.068 (0.015)	−0.057 (0.034)	−0.068 (0.015)				6.28	.00090
Anxiety					−0.10 (<0.001)	−0.086 (0.0031)		13.2	<.0001
ODD			−0.052 (0.050)		−0.074 (0.0097)			6.39	.0031
OCS				−0.047 (0.073)				3.30	.078
Somatic Problems		−0.031 (0.23)						1.45	.23
CD		−0.065 (0.020)	−0.052 (0.050)	−0.13 (<0.001)	−0.097 (0.0011)			10.2	<.0001
Sluggish Cog. Tempo		−0.086 (0.0031)		−0.057 (0.034)				7.80	.00090

Note. In all columns absent the last two, the first number listed is the Pearson’s correlation coefficient, while the value in the parentheses is the coefficient p-value. FDR correction was applied to values in the Model p-value column, and to coefficients. ODD = oppositional defiant disorder; OCP = obsessive compulsive symptoms; CD = conduct disorder; SCT = sluggish cognitive tempo.

Reward Measures and Psychopathology in Adults: In adults, scores from nine measures (TEPS, BASR, BASD, EERCT, EERCB, IGT-NE, IGTL, PDT and DDT) correlated significantly with general and specific psychopathologies; of the nine, one, EERCT, was significantly associated with the P-factor and disorder, while the remainder were unique to disorders (see Table 2). Apart from ADHD, all specific psychopathologies were significantly associated with one or more measure of reward. As with Table 1, table entries for reward measures correlating to specific psychopathologies show their Pearson’s correlation coefficient values and p-values; empty entries mean no significant relationship was found. For disorders where no relationship was found, the best performing measure was included. These findings in adults differ from the pattern seen in children, where ADHD problem severity was reward-associated but SCT, OCP, and somatic problems were not.

Table 2.

Correlation of Reward Functioning With General and Specific Psychopathologies in Adults.

Dependent variable	Independent variables
Dependent variable	Temporal exp. of pleasure	Behavioral activation system—reward resp.	Behavioral activation system—drive	Energy expenditure for reward choice total	Iowa gambling total net earnings	Probability discounting task	Energy expenditure for reward choice beta coefficient	Delayed discounting task	Iowa gambling total latency	Model F-statistic	Model p-value
P-factor				−0.065 (0.031)						5.18	.028
Depression		−0.14 (<0.001)	−0.14 (<0.001)							17.51	<.0001
Anxiety		0.10 (0.0096)	−0.0067 (0.814)	0.083 (0.0084)	−0.056 (0.055)				−0.070 (0.026)	6.73	<.0001
Avoidance Per.			−0.066 (0.030)	−0.070 (0.026)				0.065 (0.031)		5.46	.0020
ADHD			0.047 (0.11)							2.69	.10
Antisocial Per.	−0.073 (0.023)		0.17 (<0.001)							21.31	<.0001
SCT				−0.069 (0.028)	0.061 (0.041)				0.067 (0.030)	5.08	.0026
OCP			0.085 (0.0072)					0.051 (0.080)		6.60	.0024
Somatic							−0.13 (<0.001)			21.48	<.0001
Tobacco							−0.077 (0.0072)	−0.077 (0.0072)		8.38	.0048
Alcohol			0.064 (0.032)							5.00	.028
Drugs				0.086 (0.0072)						9.16	.0033

Note. In all columns absent the last two, the first number listed is the Pearson’s correlation coefficient, while the value in the parentheses is the coefficient p-value. FDR correction was applied to values in the Model p-value column, and to coefficients. Antisocial Per. = antisocial personality disorder; OCP = obsessive compulsive problems; Somatic = somatic problems; SCT = sluggish cognitive tempo.

Univariate Correlation Analysis: In both age groups, most reward measures found to be significantly associated with general and specific psychopathologies via regression also yielded statistically significant Pearson’s correlation coefficients. In children, the exceptions were: IGT-NE (ODD, CD), while in adults they were IGT-NE(Anxiety), and DDT(SCT). Overall, correlation coefficients were low, with none exceeding .2.

Discussion

We identified a general factor of psychopathology, P, that describes an individual’s overall propensity for psychopathology, by extracting the common factor between multiple disorders through PFFA. This enabled us to identify associations between measures of reward and general psychopathology, as well as specific psychopathologies.

For both children and adults, most variance in the psychopathologies assessed was explained by the P-factor, although the amount of variance explained by the P-factor in both groups differed greatly. The ~54% seen in children aligns with earlier findings but the 95% variance explained by the adult P-factor is substantially more than in previous reports (Allegrini et al., 2020; Caspi et al., 2014; Selzam et al., 2018). The large percentage of variance accounted for by the P-factor suggests that much of a presumed specific psychopathology score can be attributed to general psychopathology, rather than being regarded as reflective of disorder- specific problem severity. Simultaneously, this may have occurred because the CBCL and ASR were not designed with the intent to maximally differentiate among specific domains of psychopathology, allowing for the dominance of a general factor Importantly, removal of the SCT and OCP resulted in the P-factor explaining more variance than before in both age groups and increased loadings for all disorders. This suggests that item overlap did not falsely bolster P’s encapsulation of variance, and that scale inclusion slightly improved analysis dimensionality.

Previous work has also reported the existence of lower- order orthogonal factors, such as “internalizing” and “externalizing” disorder groupings; however, none could be identified here (Caspi et al., 2014; Lahey et al., 2014; Martel et al., 2017; Selzam et al., 2018; Waldman et al., 2016). These results may be due to the fact that previous studies incorporated different disorders from different instruments in their analyses. Additionally, earlier studies used multiple time points to derive P; repeated measurement of disorder-specific problem severity allows for better differentiation of specific factors due to reduction of state influences and noise. Also, study populations differed—it is conceivable that our sampling of children who were healthy and who had increased problems led to the strong dominance of a single factor, although the fact that the single factor was more dominant in the parent data speaks against this. Additionally, although the study population was enriched for people with mental health histories, prior literature on reward processing and psychopathology has arisen from analyses of populations that meet criteria for specific DSM disorders (Caspi et al., 2014). Factor structure results may thus differ because of the aforementioned characteristics as well as the fact that bifactor CFAs inherently assume an additional structure with specific factors, while PFFA seeks to maximize explanation of variance. Nonetheless, the low dimensionality in our data may very well be the reason why the bifactor CFA did not converge in the first place. Importantly, and in line with all of this, individual disorder loadings on our P-factor did not diverge substantially from those found in single factor CFA models (Caspi et al., 2014). Interestingly, in children, “internalizing” problem severity, for example, Depression, Anxiety, or OCP, had higher loadings on Factor 1 than externalizing disorder problem severity (ADHD, CD, or ODD). Thus, the PFFA derived factor, or P-factor, substantially captured aspects of internalizing problems compared with externalizing; this would affect the structure of any subsequent factors extracted. Furthermore, the substantial accounting of variance in both age groups by the P-factor inherently limits the explainability of subsequent factors.

The P-factor has been previously associated with various measures, including IQ, executive function, and memory (Caspi et al., 2014). In tandem, substantial work has been done on identifying linkages between different measures evaluating Reward Valuation (BAS, DDT, PDT), Reward Prediction Error (EPSC, TEPS, IGT), Responsiveness to Reward Attainment (EPSC, TEPS, IGT), and Reward Motivation/Effort Valuation (EEfRT, BASD) and various psychopathologies. This study extends existing literature in three ways: first, it identifies aberrant reward processing associated with general psychopathology in two age groups; second, it shows that some previously identified disorder-reward dysfunction correlations can be explained by the P-factor; third, it identifies areas of reward processing that appear to be disorder- specific. In children, P was significantly associated with reduced reward valuation, and with low reward motivation in adults. Both aspects of reward have been previously associated with aberrant psychiatric functioning; steep delayed discounting, identified in children, has been associated with several disorders, including ADHD, while attenuated motivation has been linked to psychiatric symptoms such as lethargy and anergy (Amlung et al., 2019; de Castro Paiva et al., 2019; Rizvi et al., 2018). In turn, these symptoms have been observed in various psychopathologies such as depression or schizophrenia (Rizvi et al., 2018; Trøstheim et al., 2020).

We found that some prior reports of disorder-specific associations could not be replicated after removing their common variance with the P-factor. This suggests that aspects of dysfunctional reward functioning previously associated with disorders were likely not specific for the disorder, but rather a consequence of comorbidity. For example, previously, steep delayed discounting was found to be linked with ADHD in children but failed to correlate with ADHD problem severity following removal of the general factor (de Castro Paiva et al., 2019). Most reward measures that had previously been associated with individual disorders were neither linked to the P-factor, nor to specific psychopathology T-scores. However, certain psychopathologies maintained previous associations following removal of general psychopathology. For example, antisocial behavior has been strongly linked with increased BAS activity and the same finding was observed following P extraction (Hoppenbrouwers et al., 2015; Murray et al., 2018). Simultaneously, SUDs have been associated with delayed discounting in the literature, but after P-factor removal, only Tobacco UD remained associated (Amlung et al., 2017).

However, even following removal of general psychopathology, most specific psychopathologies in both children and adults were associated here with unique reward measures that were not shared with the P-factor. This indicates specific areas of reward functioning particularly relevant to individual diseases. For instance, while ODD and P are both linked with increased delayed discounting, a measure of altered reward valuation, ODD is also negatively correlated with IGT-NE, a measure reflective of reduced reward responsiveness. Apart from a few disorders that did not correlate with any measures, this allows us to differentiate between mechanisms pertinent to a certain psychopathology versus a broader propensity toward general disease development. Altogether, this finding is relevant because in further studies of reward functioning and disease, it may be of interest to prioritize specific aspects of reward function relevant to unique disorders. Furthermore, understanding the interplay between general psychopathology and these disorder-specific aspects in the context of reward dysfunction may provide mechanistic insight.

Importantly, although many significant associations were found, univariate correlations between reward measures and psychopathologies (see Tables 1 and 2) were low. This suggests that while altered reward functioning is significantly linked with disorders, the magnitude of association is small. This finding is consistent with the understanding that most psychopathologies have multifactorial etiologies; other aspects such as social and cognitive functioning may account for the unobserved associations (Hess et al., 2021; Radonjić et al., 2021; Zaso et al., 2020). It is also plausible that associations between psychopathologies and the reward measures would be greater in magnitude had we been able to use direct clinical diagnoses, and acknowledge that our findings with relatively low correlations should be viewed with some caution. There is additional subjectivity as to the interpretation of the selected tasks; while they can be justified as measures of reward under the PVS, some can also be considered measures or features of decision making, in which reward is available (Cauffman et al., 2010; O’Doherty et al., 2017; Saperia et al., 2019).

Limitations

Our work must be interpreted in the context of several limitations. First, sampling for our study was enriched for psychopathology, as opposed to a population sample, and the measurement instruments used only evaluated a select group of disorders. Moreover, our population was not representative of severe psychopathology, as individuals were community-dwelling. Further study with inclusion of a greater variety of disorders and conditions that are not assessed via the ASR or CBCL, such as schizophrenia, gambling disorder, or autism spectrum disorder, as well as application of this analysis to different populations would enhance characterization of P and associated factors. Adding more disorders and more differentiated measures, possibly ones less explained by general psychopathology, would introduce greater population variance and affect computation of the P-factor and downstream calculations. Our failure to replicate previous work may also be due to our study using indirect, continuous measures of psychopathology via ASR and CBCL T-scores rather than direct clinical diagnoses.

Also, parents were the sole raters for themselves and their children; this dependency in the data opens the possibility for informant biases. Another limitation is the exclusion of children who take psychotropic medication; this limits the spectrum of psychopathology that can be captured in the sample. In tandem, we did not obtain data on traumatic stress from the participants, which has been previously associated with internalizing and externalizing psychopathologies(Carliner et al., 2016; Kevorkian et al., 2015; Marshall et al., 2016). Finally, the population demographics of participants also limit the generalizability of this study. Women are over-represented in this sample, and non-Black/African-American minorities are under-represented—additional study with a more balanced population could lead to different findings.

Conclusion

In summary, our work shows relationships between constructs of reward and general psychopathology in both children and adults, implicating different constructs across the two age groups. It also shows that some reward constructs differ in the degree to which they are associated with the P-factor and with individual psychopathologies. By identifying general and disorder-specific associations with reward functions, we may be able to separate out reward functions that play a role in the overall predisposition to psychopathology and those that steer adults or children toward a specific trajectory. Ultimately, by distinguishing between alterations in reward functioning that correspond to general versus specific psychopathologies, we can work to develop more targeted therapeutic approaches. In tandem, this knowledge can be applied to identify patients who have inclinations toward a certain psychopathology and enable early, or more efficient treatment. Altogether, this study advances knowledge about the nature of P and offers the perspective that reward mechanisms may explain the emergence of some psychopathologies and their comorbidities.

Supplemental Material

sj-docx-1-jad-10.1177_10870547231201867 – Supplemental material for Reward Functioning in General and Specific Psychopathology in Children and Adults

Supplemental material, sj-docx-1-jad-10.1177_10870547231201867 for Reward Functioning in General and Specific Psychopathology in Children and Adults by Ankita Saxena, Catharina A. Hartman, Steven D. Blatt, Wanda P. Fremont, Stephen J. Glatt, Stephen V. Faraone and Yanli Zhang-James in Journal of Attention Disorders

Footnotes

Acknowledgements

Dr. Zhang-James is supported by the European Union’s Seventh Framework Program for research, technological development, and demonstration under grant agreement no 602805 and the European Union’s Horizon 2020 research and innovation program under grant agreements No 667302. Dr. Faraone’s research has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 667302 and 965381; NIMH grants U01MH109536-01, U01AR076092-01A1, R0MH116037 and 5R01AG06495502; Oregon Health and Science University, Otsuka Pharmaceuticals and Supernus Pharmaceutical Company. Ankita Saxena is supported by the Canadian Institute of Health Research under grant agreement 202110DFD-475191-95674. We thank Patricia Forken for her role in compiling documentation for the study.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Authors: Ankita Saxena, Catharina A. Hartman, Steven D. Blatt, Wanda P. Fremont, Stephen J. Glatt, and Yanli Zhang-James declare no competing interests. In the past year, Dr. Faraone received income, potential income, travel expenses, and continuing education support and/or research support from Aardvark, Aardwolf, Akili, Atentiv, Corium, Genomind, Ironshore, Medice, Noven, Otsuka, Sandoz, Sky Therapeutics, Supernus, Tris, and Vallon. With his institution, he has US patent US20130217707 A1 for the use of sodium-hydrogen exchange inhibitors in the treatment of ADHD. In previous years, he received support from: Alcobra, Arbor, Aveksham, Axsome, CogCubed, Eli Lilly, Enzymotec, Impact, Janssen, KemPharm, Lundbeck/Takeda, Shire/Takeda, McNeil, NeuroLifeSciences, Neurovance, Novartis, Pfizer, Rhodes, Shire, and Sunovion. He also receives royalties from books published by Guilford Press: Straight Talk about Your Child’s Mental Health; Oxford University Press: Schizophrenia: The Facts; and Elsevier: ADHD: Non-Pharmacologic Interventions. In addition, he is the program director of .

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute of Mental Health [grant numbers R01MH101519-01A1 and R01MH101519-01A1S1] and the European Union’s Horizon 2020 research and innovation program for the CoCa project [grant agreement No.667302]. This report reflects only the views of the authors and the commission bears no responsibility for any uses made of the information contained in the report.

ORCID iD

Yanli Zhang-James

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Ankita Saxena, is an MD/PhD student at the department of Neuroscience at SUNY Upstate Medical University.

Catharina A. Hartman, PhD is an Associate Professor of psychiatric epidemiology at the Interdisciplinary Center of Psychopathology and Emotion Regulation (ICPE), at the University Medical Center Groningen (UMCG).

Steven Blatt, MD is a Professor at the department of Pediatrics, SUNY Upstate Medical University. He is the Medical Director of the General Pediatrics Division and the Upstate Pediatric and Adolescent Center, and the Medical Director of the Foster Care Services: ENHANCE clinic.

Wanda P. Fremont, MD is a Professor at the department of Psychiatry and Behavioral Sciences and department of Family Medicine and the Medical Director of the Child and Adolescent Psychiatry Clinic at SUNY Upstate Medical University.

Stephen Glatt, PhD is a Professor at the department of Psychiatry and Behavioral Sciences and the department of Neuroscience and Physiology at SUNY Upstate Medical University. He is the director of the PsychGene lab.

Stephen V. Faraone, PhD, is a Distinguished Professor and Vice Chair for Research at the department of Psychiatry, SUNY Upstate Medical University. He is the president of the World Federation of ADHD and a leading expert in ADHD research.

Yanli Zhang-James, MD/PhD is an associate professor at the department of Psychiatry, SUNY Upstate Medical University. Her areas of expertise include multidisciplinary research of neurosychiatric diorders and predictive modeling analytics in psychiatry.

References

Achenbach Edelbrock

. (1991). Child behavior checklist. Burlington, 7((Vt)), 371–392.

Achenbach Rescorla

. (2003). Manual for the ASEBA adult forms & profiles. University of Vermont, Research Center for Children, Youth.

Achenbach

T. M.

Bernstein

Dumenci

(2005). DSM-oriented scales and statistically based syndromes for ages 18 to 59: Linking taxonomic paradigms to facilitate multitaxonomic approaches. Journal of Personality Assessment, 84(1), 49–63. https://doi.org/10.1207/s15327752jpa8401_10

Albert

A. B.

Wagner

K. E.

Van Orman

S. E.

Anders

K. M.

Forken

P. J.

Blatt

S. D.

Fremont

W. P.

Faraone

S. V.

Glatt

S. J.

(2020). Initial responsiveness to reward attainment and psychopathology in children and adults: An RDoC study. Psychiatry Research, 289, 113021. https://doi.org/10.1016/j.psychres.2020.113021

Allegrini

A. G.

Cheesman

Rimfeld

Selzam

Pingault

Eley

T. C.

Plomin

(2020). The p factor: Genetic analyses support a general dimension of psychopathology in childhood and adolescence. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 61(1), 30–39. https://doi.org/10.1111/jcpp.13113

Amlung

Marsden

Holshausen

Morris

Patel

Vedelago

Naish

K. R.

Reed

D. D.

McCabe

R. E.

(2019). Delay discounting as a transdiagnostic process in psychiatric disorders: A meta-analysis. JAMA Psychiatry, 76(11), 1176–1186. https://doi.org/10.1001/jamapsychiatry.2019.2102

Amlung

Vedelago

Acker

Balodis

MacKillop

(2017). Steep delay discounting and addictive behavior: A meta-analysis of continuous associations. Addiction, 112(1), 51–62. https://doi.org/10.1111/add.13535

Bull

P. N.

Tippett

L. J.

Addis

D. R.

(2015). Decision making in healthy participants on the Iowa Gambling Task: New insights from an operant approach. Frontiers in Psychology, 6, 391. https://doi.org/10.3389/fpsyg.2015.00391

Carliner

Keyes

K. M.

McLaughlin

K. A.

Meyers

J. L.

Dunn

E. C.

Martins

S. S.

(2016). Childhood Trauma and illicit drug use in adolescence: A Population-Based National Comorbidity Survey Replication–Adolescent Supplement Study. Journal of the American Academy of Child and Adolescent Psychiatry, 55(8), 701–708. https://doi.org/10.1016/j.jaac.2016.05.010

10.

Caron

Rutter

(1991). Comorbidity in child psychopathology: Concepts, issues and research strategies. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 32(7), 1063–1080. https://doi.org/10.1111/j.1469-7610.1991.tb00350.x

11.

Carver

C. S.

White

T. L.

(1994). Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: The BIS/BAS Scales. Journal of Personality and Social Psychology, 67(2), 319–333. https://doi.org/10.1037/0022-3514.67.2.319

12.

Caspi

Houts

R. M.

Belsky

D. W.

Goldman-Mellor

S. J.

Harrington

Israel

Meier

M. H.

Ramrakha

Shalev

Poulton

Moffitt

T. E.

(2014). The p Factor: One General Psychopathology factor in the structure of psychiatric disorders? Clinical Psychological Science, 2(2), 119–137. https://doi.org/10.1177/2167702613497473

13.

Cauffman

Shulman

E. P.

Steinberg

Claus

Banich

M. T.

Graham

Woolard

(2010). Age differences in affective decision making as indexed by performance on the Iowa Gambling Task. Developmental Psychology, 46(1), 193–207. https://doi.org/10.1037/a0016128

14.

Chen

S. Y.

Feng

(2017). A general introduction to adjustment for multiple comparisons. Journal of Thoracic Disease, 9(6), 1725–1729. https://doi.org/10.21037/jtd.2017.05.34

15.

Cuthbert

B. N.

(2014). The RDoC framework: Facilitating transition from ICD/DSM to dimensional approaches that integrate neuroscience and psychopathology. World Psychiatry, 13(1), 28–35. https://doi.org/10.1002/wps.20087

16.

de Castro Paiva

G. C.

de Souza Costa

Malloy-Diniz

L. F.

Marques de Miranda

Jardim de Paula

. (2019). Temporal reward discounting in children with attention deficit/hyperactivity disorder (ADHD), and children with Autism Spectrum Disorder (ASD): A Systematic Review. Developmental Neuropsychology, 44(6), 468–480. https://doi.org/10.1080/87565641.2019.1667996

17.

Gard

D. E.

Gard

M. G.

Kring

A. M.

John

O. P.

(2006). Anticipatory and consummatory components of the experience of pleasure: A scale development study. Journal of Research in Personality, 40(6), 1086–1102.

18.

Garon

Moore

Waschbusch

D. A.

(2006). Decision making in children with ADHD only, ADHD-anxious/depressed, and control children using a child version of the Iowa Gambling Task. Journal of Attention Disorders, 9(4), 607–619. https://doi.org/10.1177/1087054705284501

19.

Groen

Gaastra

G. F.

Lewis-Evans

Tucha

(2013). Risky behavior in gambling tasks in individuals with ADHD–a systematic literature review. Plos One, 8(9), e74909. https://doi.org/10.1371/journal.pone.0074909

20.

Hess

J. L.

Radonjić

N. V.

Patak

Glatt

S. J.

Faraone

S. V.

(2021). Autophagy, apoptosis, and neurodevelopmental genes might underlie selective brain region vulnerability in attention-deficit/hyperactivity disorder. Molecular Psychiatry, 26, 6643–6654. https://doi.org/10.1038/s41380-020-00974-2

21.

Hoppenbrouwers

S. S.

Neumann

C. S.

Lewis

Johansson

(2015). A latent variable analysis of the Psychopathy Checklist–Revised and behavioral inhibition system/behavioral activation system factors in North American and Swedish offenders. Personality Disorders: Theory, Research, and Treatment, 6(3), 251–260. https://doi.org/10.1037/per0000115

22.

Insel

Cuthbert

Garvey

Heinssen

Pine

D. S.

Quinn

Sanislow

Wang

; National Institute of Mental Health, Bethesda, MD. (2010). Research Domain Criteria (RDoC): Toward a new classification framework for research on Mental Disorders. American Journal of Psychiatry, 167(7), 748–751. https://doi.org/10.1176/appi.ajp.2010.09091379

23.

Ivanova

M. Y.

Achenbach

T. M.

Dumenci

Rescorla

L. A.

Almqvist

Weintraub

Dobrean

(2007). Testing the 8-syndrome structure of the child behavior checklist in 30 societies. Journal of Clinical Child and Adolescent Psychology, 36(3), 405–417.

24.

Jensen

A. R.

(1993). Spearman’s g: Links between psychometrics and biology. Annals of the New York Academy of Sciences, 702, 103–129. https://doi.org/10.1111/j.1749-6632.1993.tb17244.x

25.

Kaya

Bulut

(2012). Test Review: Shipley-2 Manual. Journal of Psychoeducational Assessment, 30(6), 593–597. https://doi.org/10.1177/0734282912440852

26.

Kazdin

A. E.

(1989). Evaluation of the pleasure scale in the assessment of anhedonia in children. Journal of the American Academy of Child and Adolescent Psychiatry, 28(3), 364–372. https://doi.org/10.1097/00004583-198905000-00010

27.

Keren

O’Callaghan

Vidal-Ribas

Buzzell

G. A.

Brotman

M. A.

Leibenluft

Pan

P. M.

Meffert

Kaiser

Wolke

Pine

D. S.

Stringaris

(2018). Reward processing in depression: A conceptual and meta-analytic review across fMRI and EEG Studies. American Journal of Psychiatry, 175(11), 1111–1120. https://doi.org/10.1176/appi.ajp.2018.17101124

28.

Kevorkian

Bonn-Miller

M. O.

Belendiuk

Carney

D. M.

Roberson-Nay

Berenz

E. C.

(2015). Associations among trauma, posttraumatic stress disorder, cannabis use, and cannabis use disorder in a nationally representative epidemiologic sample. Psychology of Addictive Behaviors: Journal of the Society of Psychologists in Addictive Behaviors, 29(3), 633–638. https://doi.org/10.1037/adb0000110

29.

Kovács

Richman

M. J.

Janka

Maraz

Andó

(2017). Decision making measured by the Iowa gambling task in alcohol use disorder and gambling disorder: A systematic review and meta-analysis. Drug and Alcohol Dependence, 181, 152–161. https://doi.org/10.1016/j.drugalcdep.2017.09.023

30.

Lahey

B. B.

Zald

D. H.

Hakes

J. K.

Krueger

R. F.

Rathouz

P. J.

(2014). Patterns of heterotypic continuity associated with the cross-sectional correlational structure of prevalent mental disorders in adults. JAMA Psychiatry, 71(9), 989–996. https://doi.org/10.1001/jamapsychiatry.2014.359

31.

Marshall

D. F.

Passarotti

A. M.

Ryan

K. A.

Kamali

Saunders

E. F. H.

Pester

McInnis

M. G.

Langenecker

S. A.

(2016). Deficient inhibitory control as an outcome of childhood trauma. Psychiatry Research, 235, 7–12. https://doi.org/10.1016/j.psychres.2015.12.013

32.

Martel

M. M.

Pan

P. M.

Hoffmann

M. S.

Gadelha

Do Rosário

M. C.

Mari

J. J.

Manfro

G. G.

Miguel

E. C.

Paus

Bressan

R. A.

Rohde

L. A.

Salum

G. A.

(2017). A general psychopathology factor (P factor) in children: Structural model analysis and external validation through familial risk and child global executive function. Journal of Abnormal Psychology, 126(1), 137–148. https://doi.org/10.1037/abn0000205

33.

Murray

Shaw

D. S.

Forbes

E. E.

Hyde

L. W.

(2017). Reward-related neural correlates of antisocial behavior and callous–unemotional traits in young men. Biological Psychiatry Cognitive Neuroscience and Neuroimaging, 2(4), 346–354. https://doi.org/10.1016/j.bpsc.2017.01.009

34.

Murray

Waller

Hyde

L. W.

(2018). A systematic review examining the link between psychopathic personality traits, antisocial behavior, and neural reactivity during reward and loss processing. Personality disorders, 9(6), 497–509. https://doi.org/10.1037/per0000308

35.

Nakamura

B. J.

Ebesutani

Bernstein

Chorpita

B. F.

(2009). A psychometric analysis of the child behavior checklist DSM-oriented scales. Journal of Psychopathology and Behavioral Assessment, 31(3), 178–189.

36.

Nguyen

N. H.

Albert

A. B.

Van Orman

Forken

Blatt

S. D.

Fremont

W. P.

Faraone

S. V.

Glatt

S. J.

(2019). Effort valuation and psychopathology in children and adults. Psychologie Medicale, 49(16), 2801–2807. https://doi.org/10.1017/s0033291718003884

37.

NIMH. (2016). RDoC constructs: Domain: Positive valence systems. https://www.nimh.nih.gov/research/research-funded-by-nimh/rdoc/constructs/positive-valence-systems

38.

O’Doherty

J. P.

Cockburn

Pauli

W. M

. (2017). Learning, reward, and decision making. Annual Review of Psychology, 68, 73–100. https://doi.org/10.1146/annurev-psych-010416-044216

39.

Pulcu

Trotter

P. D.

Thomas

E. J.

McFarquhar

Juhasz

Sahakian

B. J.

Deakin

J. F. W.

Zahn

Anderson

I. M.

Elliott

(2014). Temporal discounting in major depressive disorder. Psychologie Medicale, 44(9), 1825–1834. https://doi.org/10.1017/s0033291713002584

40.

Radonjić

N. V.

Hess

J. L.

Rovira

Andreassen

Buitelaar

J. K.

Ching

C. R. K.

Franke

Hoogman

Jahanshad

McDonald

Schmaal

Sisodiya

S. M.

Stein

D. J.

van Den Heuvel

O. A.

van Erp

T. G. M.

van Rooij

Veltman

D. J.

Thompson

Faraone

S. V.

(2021). Structural brain imaging studies offer clues about the effects of the shared genetic etiology among neuropsychiatric disorders. Molecular Psychiatry, 26, 2101–2110. https://doi.org/10.1038/s41380-020-01002-z

41.

Richards

J. B.

Zhang

Mitchell

S. H.

de Wit

(1999). Delay or probability discounting in a model of impulsive behavior: Effect of alcohol. Journal of the Experimental Analysis of Behavior, 71(2), 121–143. https://doi.org/10.1901/jeab.1999.71-121

42.

Rizvi

S. J.

Lambert

Kennedy

(2018). Presentation and Neurobiology of Anhedonia in mood disorders: Commonalities and distinctions. Current Psychiatry Reports, 20(2), 13. https://doi.org/10.1007/s11920-018-0877-z

43.

Saperia

Da Silva

Siddiqui

Agid

Daskalakis

Z. J.

Ravindran

Voineskos

A. N.

Zakzanis

K. K.

Remington

Foussias

(2019). Reward-driven decision-making impairments in schizophrenia. Schizophrenia Research, 206, 277–283. https://doi.org/10.1016/j.schres.2018.11.004

44.

Scheres

Tontsch

Thoeny

A. L.

Kaczkurkin

(2010). Temporal reward discounting in attention-deficit/hyperactivity disorder: The contribution of symptom domains, reward magnitude, and session length. Biological Psychiatry, 67(7), 641–648. https://doi.org/10.1016/j.biopsych.2009.10.033

45.

Selzam

Coleman

J. R. I.

Caspi

Moffitt

T. E.

Plomin

(2018). A polygenic p factor for major psychiatric disorders. Translational Psychiatry, 8(1), 205. https://doi.org/10.1038/s41398-018-0217-4

46.

Shipley

W. C.

(1940). A self-administering scale for measuring intellectual impairment and deterioration. The Journal of Psychology, 9, 371–377. https://doi.org/10.1080/00223980.1940.9917704

47.

Siqueira

A. S. S. D.

Flaks

M. K.

Biella

M. M.

Mauer

Borges

M. K.

Aprahamian

(2018). Decision making assessed by the Iowa gambling task and major depressive disorder a systematic review. Dementia & Neuropsychologia, 12(3), 250–255. https://doi.org/10.1590/1980-57642018dn12-030005

48.

Smoller

J. W.

Andreassen

O. A.

Edenberg

H. J.

Faraone

S. V.

Glatt

S. J.

Kendler

K. S.

(2019). Psychiatric genetics and the structure of psychopathology. Molecular Psychiatry, 24(3), 409–420. https://doi.org/10.1038/s41380-017-0010-4

49.

Spearman

(1904). General Intelligence, objectively determined and measured. The American Journal of Psychology, 15(2), 201–292. https://doi.org/10.2307/1412107

50.

Stanton

McDonnell

C. G.

Hayden

E. P.

Watson

(2020). Transdiagnostic approaches to psychopathology measurement: Recommendations for measure selection, data analysis, and participant recruitment. J Abnorm Psychol, 129(1), 21–28. https://doi.org/10.1037/abn0000464

51.

Treadway

M. T.

Buckholtz

J. W.

Schwartzman

A. N.

Lambert

W. E.

Zald

D. H.

(2009). Worth the ‘EEfRT’? The effort expenditure for rewards task as an objective measure of motivation and anhedonia. Plos One, 4(8), e6598. https://doi.org/10.1371/journal.pone.0006598

52.

Trøstheim

Eikemo

Meir

Hansen

Paul

Kroll

S. L.

Garland

E. L.

Leknes

(2020). Assessment of anhedonia in adults with and without mental illness: A systematic review and meta-analysis. JAMA Network Open, 3(8), e2013233–e2013233. https://doi.org/10.1001/jamanetworkopen.2020.13233

53.

Waldman

I. D.

Poore

H. E.

van Hulle

Rathouz

P. J.

Lahey

B. B.

(2016). External validity of a hierarchical dimensional model of child and adolescent psychopathology: Tests using confirmatory factor analyses and multivariate behavior genetic analyses. Journal of Abnormal Psychology, 125(8), 1053–1066. https://doi.org/10.1037/abn0000183

54.

Zaso

M. J.

Maisto

S. A.

Glatt

S. J.

Hess

J. L.

Park

(2020). Effects of polygenic risk and perceived friends’ drinking and disruptive behavior on development of alcohol use across adolescence. Journal of Studies on Alcohol and Drugs, 81(6), 808–815. https://doi.org/10.15288/jsad.2020.81.808

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB