Cross-Cultural Psychology and the Rise of Academic Capitalism: Linguistic Changes in CCR and JCCP Articles,1970-2014

Abstract

Recently, several studies have investigated developments in academic language over the last four decades: An analysis of a large number of PubMed abstracts by Vinkers, Tijdink, and Otte showed a dramatic rise in use of positive words such as “innovative,” “robust,” “novel,” and “unprecedented.” In the field of psychology, Pritschet, Powell, and Horne found an increase in mentions of marginally significant statistical findings, and social psychologist Michael Billig raised concerns about a surge of technical jargon in the social sciences. All these phenomena are believed to be a consequence of higher publication pressure and the need to become visible as a researcher in an increasingly competitive climate that is often referred to as academic capitalism. In our study, we tested the aforementioned indicators of linguistic change for a sample of 1,680 research articles from the Journal of Cross-Cultural Psychology (JCCP) and 657 research articles from Cross-Cultural Research (CCR), published between 1970 and 2014. Overall, we found a consistent increase in positive framing, a rise in reports of marginally significant statistical findings, and indicators for growth in technical jargon. These findings suggest that self-marketing strategies are also on the rise in cross-cultural psychology.

Keywords

linguistic change cross-cultural psychology academic capitalism publish or perish positive framing

The limits of my language mean the limits of my world.

—Wittgenstein (1922, p. 86).

Science is a part of the social world and the social world undergoes changes. For the last 250 years, there has been an exponential growth of scientific publications, with up to approximately 2,000,000 scientific publications in the year 2012 alone (Sarewitz, 2016). The most dramatic changes in academic publishing happened in the 20th century when industrial societies were transformed into information societies (Castells, 2010). Increasingly, universities found themselves competing for students and financial resources, and the amount of scientific publications emerged as a preeminent criterion of individual researchers’ and academic institutions’ quality. Whereas in 1969 only 50% of academics in U.S.-American universities had published during the previous 2 years at all, this figure had already risen to two thirds by the 1990s (Billig, 2013). Publishing in prestigious academic journals was also no longer a privilege solely for researchers at a few distinguished universities, but became a requirement for college professors and academics outside the United States and Western Europe as well (for an overview, see Billig, 2013, p. 11ff). The advent of the Internet as a mass phenomenon and the increasing number of open access journals may nowadays even accelerate the increase in academic publications per year. Apart from publish or perish, get visible or vanish seems to be a further trend of the day: It is not only important to publish a lot, but one’s research must attract the attention of the scientific community as well as of the general public (Doyle & Cuthill, 2015). The term academic capitalism is now often used to compare the increasing struggle for excellence in a globalized academic world with the fierce competition in boundless economic markets (Münch, 2014; Schneickert & Lenger, 2016; Slaughter & Rhoades, 2004).

Over the last years, several studies have investigated whether these changes in the organizational foundations of science have also led to changes in the language of academic publications in the form of (a) increasingly positive framing, indicated by the use of a higher number of words with positive connotations accompanied by exaggeration of the importance of research findings; (b) increasing willingness of scientists to report rather small and potentially uncertain statistical effects as marginally significant; and (c) increasing use of technical jargon and overly complicated language. We argue that as part of the general trend toward more competition in academia, self-marketing has increasingly become the inherent logic of academic writing also in the field of cross-cultural psychology over the last decades.

Previous Studies

Positive Framing

The Pollyanna hypothesis states that there is a universal human tendency to use positive words more frequently, diversely, and facilely than negative words (Boucher & Osgood, 1969). This tendency has been confirmed in large text corpora and for different languages (Garcia, Garas, & Schweitzer, 2012). However, over and above this general preference for positive words—which should be time-invariant—longitudinal changes have been observed in the relative frequency of positive word usage. Linguistic analyses show that the language style of scientific publications in medicine and related fields has changed substantially over the last four decades. For example, an analysis of all PubMed abstracts from the years 1974 to 2014 revealed an increase in usage of 880% for a selection of 25 positively connotated words. This increase has been driven in large part by the more frequent use of words such as “robust,” “unprecedented,” and “innovative.” The frequency of negatively connotated words increased significantly as well, but to a smaller degree (Vinkers, Tijdink, & Otte, 2015). This finding is widely interpreted to be the result of a marked rise in the use of advertising language in scientific publications thought to be due to higher publication pressure as well as increasing reliance on external funding (Moore, Neylon, Eve, O’Donnell, & Pattinson, 2016).

In contrast to this dramatic increase of such language in medical literature, the picture for written expressions in general is ambiguous. For the Google Ngram corpus¹ of 8 million digitized English language books, linguistic analyses revealed a general decrease in the use of emotion-related words throughout the 20th century and periods of positive and negative affect (Acerbi, Lampos, & Bentley, 2013). For song lyrics, a decrease in their happiness from the 1960s to the 1990s was shown, while for blogs, an increase in happiness from 2005 to 2009 was found (Dodds & Danforth, 2010).

Highlighting the Significance of Findings Even When They Are Marginal

Pritschet, Powell, and Horne (2016) have shown, based on an analysis of over 1,500 articles from top-tier psychology journals, that the practice of describing p values as “marginally significant” became more common in cognitive psychology, developmental psychology, and social psychology between 1970 and 2010, with a peak in 2000 in the latter two subdisciplines. The slight decrease since then could be a consequence of the mounting criticism of the use of significance tests in psychology (e.g., Gigerenzer, 2004) and the emergence of Bayesian statistical methods (e.g., Gelman, Carlin, Stern, & Rubin, 2014). In view of the current “crisis of confidence” (Pashler & Wagenmakers, 2012) or “reproducibility crisis” (Open Science Collaboration, 2015), p-hacking—the practice of using questionable research practices to achieve statistical significance to increase the chances of getting published—has become a topic of intense debate, particularly in the field of social psychology. A generous use of the label marginally significant for weak effects—at least in cases where they support the researcher’s hypothesis—corresponds well with the general picture of increasing pressure to advertise one’s research. Under such circumstances, researchers would be all ready to find their hypotheses confirmed.

Technical Jargon and Nominalization

In 2013, Michael Billig summarized two decades of research into academic jargon in such diverse fields as discourse analysis and mainstream experimental social psychology in his book Learn to Write Badly: How to Succeed in the Social Sciences. His main argument is that increasing publication pressure and the vast number of scientific publications lead to the development of more and more disintegrated scientific subcommunities. Such groups use their own jargon and their own increasingly complex and complicated idiosyncratic terminology to set themselves apart from other communities, who may investigate the very same phenomena using the very same research methods.

One consequence is an increasing nominalization of scientific language: processes or series of events are turned into things. For example, “It is simpler to use ‘deindividuation,’ rather than repeatedly employ a clause such as ‘being in a state of mind of not feeling oneself to be an identifiable individual’” (Billig, 2011, p. 11). Of course, this is not bad in itself, but it should be noted that nominalization leads to a more imprecise language, although it may sound more technical and precise: The term deindividuation does, for example, not entail the information as to who actually does not feel identifiable anymore. Through nominalization and the related passivization, or writing about processes in the passive voice, scientific texts are depersonalized: People and actions are removed from psychological analyses, although they are (or should be) the central subject of investigation. Even worse, these fictional things, which are nothing more than descriptions of processes, can take on a life of their own and become the guiding idols of these scientific subcommunities.

Apart from publish or perish, academic research depends more and more on external funding as well. According to Billig, one consequence of this increasing need to attract funding is a more frequent use of the claim that more research (=more money) is needed to pursue research questions that were opened up through an empirical study (Billig, 2013).

Research Questions and Rationale

In the present study, we investigated to what degree cross-cultural psychology as a field of study was affected by the changes mentioned above. In so doing, we also intended to contribute to the ongoing self-reflection among scholars in general, and in the Journal of Cross-Cultural Psychology (JCCP) in particular (cf. Allik, 2013; Best & Everett, 2010; Brouwers, Van Hemert, Breugelmans, & Van de Vijver, 2004; van de Vijver & Lonner, 1995). The investigation was carried out in direct comparison with Cross-Cultural Research (CCR), another multidisciplinary journal that publishes comparative studies.

Despite the fact that several textbooks discuss the diversification of cross-cultural psychological topics during the past several decades (e.g., Smith, Bond, & Kagitcibasi, 2006), no empirical research has yet investigated similarities and differences in terms of topics and linguistic features between these two journals. Consequently, it is currently unknown whether the distinct content of JCCP and CCR or changes in their topic preferences relate to general linguistic trends displayed in their published articles.

To provide a context for our study through a preliminary comparative linguistic examination of our research material, we start by identifying the words that show the clearest differentiation between the JCCP and CCR articles and examine general topic-related changes within the two journals between 1970 and 2014. We then continue with the main analysis, investigating (a) whether there was substantial change in the expression of positive and negative emotions; (b) whether the frequency of the use of terms that indicate marginal significance increased; and finally (c) whether the use of words made up of more than six letters rose, whether the use of common verbs decreased, whether the frequency of nominalizations expanded, and whether the claim that “more research is needed” was expressed more frequently.

Our methodological approach differs to some degree from the aforementioned previous studies. Vinkers et al. (2015) had used self-constructed word-category scales comprising no more than 25 words and they only analyzed the abstracts of scientific publications. In our analysis, we used these self-created dictionaries in direct comparison with the positive emotions and negative emotions scales of an established psycho-linguistic dictionary—the LIWC (Linguistic Inquiry and Word Count) 2015 internal dictionary. Using the Vinkers et al. measure (25 positive words) comparatively with the LIWC dictionary (625 positive words) allowed us to establish whether upward trends in positive word usage were restricted to a small number of specific buzzwords or were part of a much broader linguistic base. Furthermore, we went beyond the scope of their work by analyzing not only abstracts but also the entire bodies of the CCR and JCCP articles’ texts.

Pritschet et al. (2016) had manually coded their research articles for instances of reports of marginally significant effects. In our study, we decided to use an automated linguistic approach based on a word category scale comprised of synonyms and indicators of marginally significant statistical effects (see below). Although this approach has its limitations when it comes to accuracy (see the “Discussion” section), it allows for an automatized analysis of higher numbers of publications.

Michael Billig’s (2013) critique is on one hand based on a qualitative in-depth analysis of a small number of academic publications. On the other hand, he bases his claims on studies which focused on comparing academic writing in textbooks (Biber, Conrad, & Cortes, 2004) or a very broad range of scientific articles from different disciplines (Biber & Gray, 2010) with everyday language. Our study focused on longitudinal developments over a selected time period for a specific field of study: cross-cultural psychology.

Method

General Approach

The main focus of this article is on the linguistic changes in JCCP and CCR articles between 1970 and 2014. As a means of establishing a context for our findings, we also included analyses of general linguistic differences between JCCP and CCR and of general linguistic trends over the given time period.

Contextualization

We used the AntConc software (Anthony, 2005) to identify the words that predicted best whether an article was from the JCCP or the CCR corpus using a comparative linguistic approach. The software compares two (or more) text corpora and identifies the words that have the highest keyness or ability to differentiate between the corpora. As a means of establishing a context for our findings, we also examined general linguistic trends and topic-related changes for the two journals by identifying keywords for the respective articles from different time periods, again, using the AntConc software. For this analysis, the data were split into three time periods, namely, 1970-1984, 1985-1999, and 2000-2014. In specific, this analysis identified the words that differentiated most clearly between one of the time periods and the other two time periods. It thus also allowed for illustrating changes in the research focus of these two journals over time.

Analysis of Linguistic Changes

We used the LIWC 2015 Software (see Pennebaker, Boyd, Jordan, & Blackburn, 2015) to analyze the prevalence of words within a given category in comparison with the total number of words within a document. The LIWC software originated from a software tool that was developed as part of a research project on language and self-disclosure (Pennebaker, 1993). The current version LIWC 2015 is a proprietary analysis software, which, like its predecessor LIWC 2008, has been and is being used in a large number of scientific and commercial studies. Word categories were created to measure an underlying abstract construct in analogy to a psychometric scale (Pennebaker, 1993). The approximately 90 (depending on the exact version) LIWC scales are the result of a thorough and elaborate construction process that comprises not only expert ratings of the goodness of fit of a word for a given scale but also statistical evaluations of the scales’ psychometric properties (Pennebaker et al., 2015,). A closer description of all research material and word category scales we used can be found below.

Trend analysis

For a first visual inspection of the developments over time, we show trends graphically using local polynomial smoothing via the lpoly command in Stata 12. This technique performs an Epanechnikov kernel-weighted local polynomial regression and displays a graph of the smoothed values with a 95% confidence interval. Local polynomial smoothing is a nonparametric modeling approach, which means that instead of assuming a certain functional fit (e.g., linear, exponential, etc.) ex ante, the data “speaks for itself” (Gutierrez, Linhart, & Pitblado, 2003). This property is thus beneficial for exploratory purposes where we want to first smooth the data without imposing assumptions about their distribution. The smoothing itself is reasonable, as the papers we chose might well have been published a year earlier or later depending on arbitrary delays in the review process. The graphical analyses also allow us to identify nonlinear effects such as changes within a certain time period during the sampling frame. Furthermore, the graphical analyses allow for a direct comparison of the developments in the JCCP and in the CCR articles.

As the local polynomial smoothing reveals trends that often appear monotonic (but not necessarily linear), we used Spearman’s rank correlation coefficient r_s to assess the strength and statistical significance of these monotonic changes. To additionally get an impression of what these up- and downward changes meant in substantial terms, we compared the mean relative frequency of the phenomenon under study (e.g., positive words) in the early years (1970-1984) with the mean relative frequency of the phenomenon in the later years (2000-2014) via unpaired t tests (which were, for instance, also used by Vinkers et al., 2015). This division of the time frame under study corresponds to the one used for the contextualization analysis (see above). Taken together, these measures allowed us to gain a comprehensive picture of the linguistic trends observed.

Data

We analyzed publications classified as “research articles” that were published between 1970 and 2014 in JCCP (1,680 articles) and CCR (657 articles). Overall, more than 15 million words were processed.

The articles were split into an abstract and a text body section, with the latter comprising the main document without tables, figures, references, and appendices. For the analysis of positive and negative emotions, we started with an analysis of the abstracts. On one hand, this step enabled us to compare our findings with the results obtained in other fields of research (e.g., Vinkers et al., 2015). On the other hand, it is also important in its own right because the abstracts constitute the information about a study that is often most accessible to the wider public via catalogues such as EBSCO or search engines such as Google Scholar, which makes them particularly interesting as a marketing tool. Next, we addressed the question of whether the rise of positive words only applied to the abstracts or whether all of the academic language as represented by the text bodies was affected as well. If an increase in the use of marketing language also occurred in the text bodies, this would indicate that the changes in academic writing do not just affect the surface of academic publications (i.e., the titles and abstracts), but go much deeper. For the remaining analyses, only the text bodies were used.

Scales From Vinkers et al. (2015)

Positive 25 and Negative 25

In line with the previously mentioned study by Vinkers et al. (2015), we used the original 25 positive words (“Positive 25”; for example, “amazing,” “spectacular”) and the original 25 negative words (“Negative 25”; for example, “disappointing,” “unacceptable”) that had been used in their study. The complete word lists can be found in the original publication. We decided to leave out the 25 neutral words, because most of them appeared to be closely related to medical topics (e.g., “blood,” “bone,” “disease”).

Top four positive words

We also created a category scale comprising the four words “robust,” “novel,” “innovative,” and “unprecedented,” whose frequencies increased particularly dramatically in the study by Vinkers et al. (2015 p. 1).

Scales From the Internal LIWC 2015 Dictionary

LIWC positive emotions

This scale is comprised of 620 words. Examples are “love,” “nice,” and “sweet” (Pennebaker et al., 2015, p. 3). Its corrected internal consistency is .64.

LIWC negative emotions

This scale is made up of 744 words. Examples are “hurt,” “ugly,” and “nasty” (Pennebaker et al., 2015, p. 3). Its corrected internal consistency is .55.

LIWC verbs

This scale comprises 1,000 verb forms of common verbs. Examples are “eat,” “come,” and “carry” (Pennebaker et al., 2015, p. 3).

LIWC six-letter words

All words comprised of at least six letters were counted here. This measure has been used as an indicator of text difficulty in numerous studies in the tradition of Flesch (1948).

Self-Constructed Scales

Marginal significance

We constructed this scale from different expressions that are related to the concept of marginally significant statistical effects (“marginally significant,” “p < .10,” “p<.10,” “p .10,” “marginal significant,” “marginal significance,” “trend towards significance,” “trended towards significance,” “trending towards significance,” “approaching significance,” “approached significance”).

Nominalizations

For this scale, we first made a list of all 4,235 words ending with the suffixes “-ation,” “-ity,” and “-ment.” A student assistant then removed all entries that were obviously not related to Billig’s use of the term nominalization (such as “city,” “personality,” “education,” “development”). After further revision through the authors, we finally settled on a list of 243 words. The full list can be obtained from the authors.

Results

Unique Aspects of JCCP and CCR Articles

A comparison of the 25 words with the highest keyness (i.e., the best predictors of whether an article was from the JCCP or the CCR corpus; Table 1) showed that in the JCCP articles, terms from the field of psychology such as “personality,” “values,” and “acculturation” had been used more frequently than in the CCR articles. On a similar note, statistical terms such as “items” and “subjects” were typical for JCCP articles, as well as words indicating a research focus on East Asia such as “Chinese,” “Japanese,” and “Hong Kong” (presumably being compared with U.S.-Americans and other Westerners). The term “differences” was more frequently used in the JCCP articles. In the case of the CCR articles, words that are related to a more humanities-oriented approach such as “anthropology,” “ethnology,” and “evolution” were more frequent. Also, terms referring to databases, such as “Ethnographic Atlas,” “Standard Cross-Cultural Sample (SCCS),” and “Human Relations Area Files (HRAF),” or specific methodological terms, for example, “(spatial) autocorrelation,” were more typical for CCR articles. Similarly, there was a clear difference between the CCR and JCCP articles in the use of words that represent cultural practices such as “polygyny,” “subsistence,” “marriage,” “codes,” or “warfare.” Taken together, JCCP represented a more psychology-oriented quantitative approach toward cross-cultural research, whereas CCR seemed to be leaning more toward a descriptive humanities-oriented approach.

Table 1.

Keyword Analysis of Linguistic Differences Between the JCCP and the CCR Articles.

No.	JCCP articles			CCR articles
No.	Frequency	Keyness	Word	Frequency	Keyness	Word
1	33,285	7.78	self	9,583	9.74	societies
2	15,869	6.14	chinese	1,741	3.81	hraf
3	22,286	4.88	journal	2,216	3.72	ethnographic
4	14,299	4.26	participants	2,540	3.20	anthropology
5	24,978	3.93	differences	2,940	3.03	sexual
6	9,428	3.81	japanese	1,527	2.94	warfare
7	9,586	3.46	subjects	2,912	2.84	acceptance
8	24,212	3.42	group	1,219	2.60	polygyny
9	15,662	3.13	items	1,208	2.17	files
10	9,214	3.00	americans	2,750	2.06	cases
11	26,042	2.96	children	764	1.82	sccs
12	6,775	2.84	acculturation	1,384	1.79	evolution
13	12,537	2.74	personality	2,265	1.77	marriage
14	57,673	2.50	cultural	1,552	1.75	labor
15	18,430	2.50	values	1,331	1.74	codes
16	25,571	2.28	culture	1,112	1.69	subsistence
17	10,566	2.22	ethnic	1,835	1.61	color
18	7,905	2.13	item	691	1.59	autocorrelation
19	7,710	1.90	english	5,758	1.56	women
20	19,978	1.86	american	834	1.56	ethnology
21	3,867	1.84	hong	861	1.46	agriculture
22	13,315	1.82	students	3,067	1.46	political
23	12,452	1.74	across	1,134	1.38	kin
24	3,449	1.64	kong	938	1.36	kinship
25	5,508	1.63	performance	962	1.35	anthropologist

Note. AntConc v 3.4.4 (Anthony, 2005) was used for the analysis; in the table, functional words, author names, and other words and expressions without any obvious relation to contents of the journal were omitted; the complete list is available here https://osf.io/6unv4/ . JCCP = Journal of Cross-Cultural Psychology; CCR = Cross-Cultural Research; Frequency = frequency of the word within the respective corpus; Keyness = log-likelihood-ratio of the word’s statistical specificity for the respective corpus (cf. Kilgarriff, 2001); HRAF = Human Relations Area Files; SCCS = Standard Cross-Cultural Sample.

General Linguistic and Topic-Related Changes in the Two Journals Over Time

Next, we examined general linguistic trends for the two journals by identifying keywords for the respective articles from the time periods 1970-1984, 1985-1999, and 2000-2014. See Table 2 for developments within the JCCP articles and Table 3 for developments within the CCR articles.

Table 2.

Keyword Analysis for JCCP Articles From 1970-1984, 1985-1999, and 2000-2014.

No.	1970-1984			1985-1999			2000-2014
No.	Frequency	Keyness	Word	Frequency	Keyness	Word	Frequency	Keyness	Word
1	4,612	3,589.00	subjects	4,278	1,580.29	subjects	11,900	11,502.97	participants
2	8,033	1,527.99	children	884	1,437.76	television	17,153	5,231.27	we
3	818	1,167.58	perceptual	10,618	1,036.98	psychology	21,818	4,585.64	self
4	1,413	1,155.42	mexican	10,416	977.96	cross	12,787	3,870.57	values
5	382	828.98	illusion	4,955	937.69	students	8,430	3,580.45	model
6	1,274	812.32	subject	927	755.08	risk	2,304	2,432.01	parenting
7	734	804.52	conservation	266	569.36	songs	8,307	2,036.72	across
8	461	780.40	piaget	1,628	552.92	triandis	2,764	1,993.26	fit
9	792	773.23	modern	186	494.15	fitd	1,812	1,925.96	construal
10	1,517	773.08	males	323	461.56	mmpi	1,793	1,890.03	relational
11	2,477	754.05	sex	261	452.96	inuit	2,353	1,851.01	schwartz
12	1,463	746.40	white	3,299	445.87	japanese	5,349	1,750.13	country
13	358	696.13	pictorial	546	418.03	filipino	4,718	1,698.21	acculturation
14	1,178	689.29	indian	245	393.40	silence	841	1,465.76	children
15	394	632.77	modernity	427	373.89	hui	692	1,436.32	nso
16	1,379	625.77	females	5,602	359.69	university	748	1,425.15	rmsea
17	2,197	616.52	responses	171	346.50	embarrassability	1,069	1,421.06	honor
18	1,379	625.77	females	426	341.29	attractiveness	1,367	1,411.58	invariance
19	953	607.27	grade	282	338.40	safety	1,445	1,406.84	russian
20	1,046	598.51	black	400	335.20	etic	2,213	1,384.37	contexts
21	3,599	587.14	age	290	332.47	philippine	1,257	1,355.03	openness
22	2,517	568.08	school	441	324.10	emic	2,130	1,283.88	autonomy
23	888	567.67	field	165	316.43	harassment	5,108	1,275.11	individuals
24	500	557.43	color	288	300.67	nuclear	987	1,255.58	relatedness
25	686	557.10	chicano	364	287.50	disclosure	2,518	1,226.76	satisfaction

Note. AntConc v 3.4.4 (Anthony, 2005) was used for the analysis; in the table, functional words and other words and expressions without any obvious relation to contents of the journal were omitted; the complete list is available here https://osf.io/3yx69/ . JCCP = Journal of Cross-Cultural Psychology; Frequency = Frequency of the word within the respective corpus; Keyness = log-likelihood-ratio of the word’s statistical specificity for the respective corpus (cf. Kilgarriff, 2001).

Table 3.

Keyword Analysis for CCR Articles From 1970-1984, 1985-1999, and 2000-2014.

No.	1970-1984			1985-1999			2000-2014
No.	Frequency	Keyness	Word	Frequency	Keyness	Word	Frequency	Keyness	Word
1	1,161	1,147.40	naroll	1,504	3,172.75	moral	2,421	1,434.60	acceptance
2	633	1,059.60	diffusion	1,529	1057.56	murdock	1,337	1,397.50	adjustment
3	347	1,031.87	shetland	349	927.27	kohlberg	912	804.74	paternal
4	1,165	909.89	problem	530	759.86	subjects	2,174	802.83	countries
5	858	806.64	warfare	263	707.67	cassava	1,518	712.34	perceived
6	559	788.17	galton	299	672.81	famine	1,979	698.80	parental
7	398	735.28	raoul	260	608.73	bone	813	650.12	maternal
8	244	725.58	orkney	500	608.56	justice	936	644.36	punishment
9	864	724.29	hraf	412	590.28	homosexuality	470	636.55	corporal
10	707	716.13	paper	3,544	589.30	social	1,420	589.73	color
11	1,387	689.68	test	337	570.95	envy	917	527.85	fathers
12	747	672.14	residence	3,819	565.64	cross	1,087	517.44	participants
13	1,066	670.09	correlation	364	557.65	jealousy	1,868	448.98	psychological
14	201	597.71	habs	545	553.85	stage	466	436.56	barber
15	362	572.78	matrilocal	454	528.56	property	352	434.64	teen
16	216	554.45	cargo	5,344	526.68	cultural	2,653	395.50	self
17	999	554.36	ethnographic	556	491.22	roles	910	385.58	mothers
18	3,143	547.05	societies	700	466.65	codes	1,042	378.80	rohner
19	563	523.26	bias	324	456.30	reasoning	1,648	378.70	national
20	199	505.81	film	201	448.33	deterrence	336	373.20	happiness
21	348	504.19	taboo	215	439.29	dwelling	387	372.47	trust
22	221	496.44	freud	3,284	438.41	societies	264	370.53	khaleque
23	576	475.77	percent	184	437.72	starvation	258	362.11	inglehart
24	491	475.56	atlas	228	396.79	rape	1,382	360.17	parents
25	170	462,68	misses	264	362.26	private	1,139	344.58	country

Note. AntConc v 3.4.4 (Anthony, 2005) was used for the analysis; in the table, functional words and other words and expressions without any obvious relation to contents of the journal were omitted; the complete list is available here https://osf.io/r6g4v/ . CCR = Cross-Cultural Research; Frequency = Frequency of the word within the respective corpus; Keyness = log-likelihood-ratio of the word’s statistical specificity for the respective corpus (cf. Kilgarriff, 2001); HRAF = Human Relations Area Files; HABS = HRAF Automated Bibliographic System.

JCCP

While the term “subject(s)” was used frequently until 1999, authors referred to “participants” or “individuals” in more recent articles. The increasing use of “we” seems to indicate that, from about the year 2000 on, more papers were authored by more than one researcher, compared with earlier decades. An alternative explanation might be that the practice of using “we” to include the reader even when a paper is single-authored has become more common.

The origin of samples (Mexican → Japanese → Nso) and the most cited researchers (Piaget → Triandis → Schwartz) changed over time as well. We also observed changes in topic preferences: JCCP developed from a journal with a developmental focus (1970-1984: “children,” “grades,” and “schools”), over a period in which cultural uniqueness was emphasized (1985-1999: “etic,” “emic,” “Inuit”), into an outlet with a focus on statistical modeling approaches (2000-2014: “RMSEA” [root mean square error of approximation] and “fit”). These changes in research methods might also indicate a shift in JCCP’s focus from cross-cultural differences to similarities (e.g., “invariance”).

The analysis also sheds light on the topics that were in the focus of research during the respective time periods: Understanding cultural differences in “perceptual illusions” and Piaget’s “conservation task” marked the beginning of modern cross-cultural psychology (≤1984), followed by a period of research on cultural differences in the tolerance of silence and personality (1985-1999; for example, the Minnesota Multiphasic Personality Inventory [MMPI]). Since 2000, a small number of theoretical models seem to have dominated the field (self-construals, acculturation, and autonomy vs. relatedness; for a review, see Smith et al., 2006). Some traditional topics, such as child development (“parenting” and “children”), regained the interests of researchers publishing in JCCP, while we also observed the emergence of new topics, such as, “honor,” “values,” and “satisfaction.”

CCR

Until 1984, the focus of CCR was on “societies,” whereas “subjects” or “participants” were studied more frequently in more recent articles. From 2000 to 2014, differences between societies (“national,” “countries,” “country”) were again increasingly emphasized.

We found for the CCR as well that each time period represented the legacy of only a few outstanding researchers (Naroll → Murdock → Barber). In terms of topic-related changes, from 1970 to 1984, the focus was on data from ethnographic datasets (HRAF, HABS [HRAF Automated Bibliographic System], Ethnographic Atlas). Researchers frequently debated critical issues in their field (“Galton’s problem,” “autocorrelations,” “bias”), and the topics of studies were phenomena such as matrilocal residence and warfare between groups. From 1985 to 1999, a number of new topics came into fashion which were related to “moral,” “justice,” “homosexuality,” and scarcity of resources (“famine,” “starvation,” “envy”). Since 2000, terms associated with child development and parental practices had high “keyness” in the CCR corpus (“acceptance,” “adjustment,” and “paternal”). These topic preferences seemed to go along with the increased description of mental processes (“self” and “psychological”), and, similar to JCCP (see above, “satisfaction”), the emergence of positive psychology (e.g., “happiness”) as a new field of interest for cross-cultural researchers.

The Expression of Positive and Negative Emotions

A visual inspection of the first row of smoothed graphs in Figure 1 shows that across all measures, in both journals, and both in the abstracts and text bodies, the frequency of positive words appears to increase. The extent of the increase, however, differs. The trends in the usage of negative words, by contrast, are much more ambiguous with a mixture of increases, decreases, and reverse u-shaped patterns.

Figure 1.

The relative frequency of positive and negative words, 1970-2014: (a) Abstracts and (b) Full texts.

Analysis of the abstracts

The relative frequency of the 25 positive words from Vinkers et al. (2015) increased significantly in both JCCP (r_s = .05, p < .05) and CCR (r_s = .15, p < .001) abstracts. Although the increase is stronger in CCR, it should be noted that in substantial terms, CCR started from an exceptionally low initial level with virtually no mentions of these 25 positive words in the first 15 years under study (M = 0.00), rising significantly, t(426) = 2.68, p < .01, to a relative frequency of on average 0.04% of all words in the last 15 years (for additional figures such as standard deviations, see Tables A2 and A3 in the appendix). JCCP by contrast had a higher initial starting point (M = 0.04%) and remained above CCR in the end (M = 0.05%) despite the lower increase, t(1230) = 0.93, p = .35 (Figure 1a).

The LIWC Positive Emotions scale also shows significant monotonic increases over time in both JCCP (r_s = .16, p < .001) and CCR abstracts (r_s = .31, p < .001). Again, we found a stronger increase for the CCR abstracts, with a rise from an average 1.69% to an average 2.88%, t(426) = 5.70, p < .001. In JCCP, the frequency of positive emotions increased from a mean of 1.91% to a mean of 2.87%, t(1230) = 7.35, p < .001 (Figure 1a).

Regarding negative emotions, CCR abstracts showed no sign of monotonic change using the 25 negative words scale by Vinkers et al. (2015; r_s = .00, p = 1) or the LIWC negative emotions scale (r_s = −.07, p = .11). In JCCP, however, there appeared to be a slight increase at least with regard to the broader set of negative emotions in LIWC (r_s = .05, p < .05). For the 25 negative words by Vinkers et al., this increase did not reach statistical significance (r_s = .04, p = .086). A closer look at the CCR abstracts revealed an unusually high number of negative emotions (LIWC) in the late 1980s, which might be related to the specific research focus of this journal in this period, which was characterized by a high frequency of negatively connoted key terms such as “famine,” “envy,” “jealousy,” “deterrence,” “starvation,” and “rape” (see Table 3).

Analysis of the text bodies

In the text bodies (Figure 1b), the increase in expressions of positive emotions seemed to be more substantial and more homogeneous in both journals than it was for the abstracts. The usage of the 25 positive words identified by Vinkers et al. (2015) increased significantly in both CCR (r_s = .22, p < .001) and JCCP (r_s = .23, p < .001) between 1970 and 2014. In substantial terms, the relative frequency doubled in both journals from an average 0.02% in the first 15 years to 0.04% in the last 15 years—CCR: t(487) = 3.52, p < .001; JCCP: t(1223) = 4.54, p < .001. The effect is even stronger for LIWC’s broader set of positive emotions (CCR: r_s = .36, p < .001; JCCP: r_s = .31, p < .001). Here, the relative frequency of positive emotions increased by .74 percentage points from 1.65% to 2.39%, t(1223) = 11.61, p < .001, in JCCP text bodies and .61 percentage points from 1.55% to 2.16%, t(487) = 6.98, p < .001 in CCR text bodies.

In addition, we looked at the four positive words that had the greatest impact in the study by Vinkers et al. (2015; “robust,” “novel,” “innovative,” and “unprecedented”). Again, we found significant upward trends (CCR: r_s = .22, p < .001; JCCP: r_s = .27, p < .001), and the relative change in the use of these four words seemed to be larger than it was for the other two measures for positively connotated words. The relative frequency of these four words tripled in JCCP papers, rising from 0.003% to 0.009%, t(1223) = 5.27, p < .001, and quadrupled in CCR articles, rising from 0.003% to 0.012%, t(487) = 2.14, p < .05. Note that an equivalent analysis for the abstracts could not be conducted as almost 99% of all abstracts did not contain any of the top four positive words, leaving too little variance for meaningful trends to be observed. This lack of usage in the abstracts provides further evidence that the abstracts did not have a special “advertising” function. The linguistic change toward more positive words also affected the full texts, at least in cross-cultural psychology.

By contrast, almost no meaningful changes in the expression of negative emotions could be observed in the text bodies. Neither the LIWC (r_s = −.04, p = .31) nor the Vinkers et al. measure (r_s = −.04, p = .29) of negative words changed significantly over time in CCR articles. In JCCP, the Vinkers et al. measure also did not show any effect (r_s = −.00, p = .90), while the LIWC scale indicated a minor upward move (r_s = .06, p < .05).

A further observation is that in line with the Pollyanna hypothesis, using the respective LIWC dictionaries, positively connotated words occurred on average more frequently than negatively connotated words in both JCCP abstracts (2.43% positive emotions vs. 1.26% negative emotions), t(1679) = 16.73; p < .001, and JCCP text bodies (2.04% positive emotions vs. 1.09% negative emotions), t(1685) = 27.04; p < .001, as well as CCR abstracts (2.32% positive emotions vs. 1.47% negative emotions), t(565) = 6.65; p < .001, and text bodies (1.82% positive emotions vs. 1.22% negative emotions), t(715) = 12.16; p < .001.²

The Frequency of Articles Reporting Marginally Significant Effects

In line with our expectations, the number of articles mentioning marginally significant effects increased over time (Figure 2). This growth trend is significant in both CCR (r_s = .14, p < .001) and JCCP (r_s = .11, p < .001) articles. The percentage share of articles mentioning marginally significant effects almost doubled in JCCP articles (from 10.6% to 20.0%), t(1223) = 4.36, p < .001, and almost quadrupled in CCR articles (from 3.7% to 14.7%), t(487) = 4.13, p < .001, between the first and the last 15 years under study.

Figure 2.

The relative frequency of mentions of marginal significance, 1970-2014.

The Frequencies of Words With Six or More Letters, Common Verbs, Nominalizations, and the Claim That “More Research Is Needed”

The frequency of long words (six or more letters) increased in both journals over time (upper left-hand graph in Figure 3). The increase was stronger in JCCP (r_s = .39, p < .001) than in CCR (r_s = .23, p < .001). In substantive terms, the increase was 3.2 percentage points (from 29.6% to 32.8%) between the first and the last 15 years under study in JCCP, t(1223) = 18.74, p < .001, and 1.3 percentage points (30.3% to 31.6%) in CCR, t(487) = 4.08, p < .001.

Figure 3.

The relative frequency of long words, common verbs, nominalizations, and “more research is needed,” 1970-2014.

In line with the previous finding, the frequency of common verbs (upper right-hand graph in Figure 3) decreased significantly in both journals. The trend was stronger in CCR (r_s = −.32, p < .001) than in JCCP (r_s = −.08, p < .001). The frequency of common verbs dropped by 6.1% in JCCP articles, t(1223) = −5.98, p < .001, and 13.1% in CCR articles, t(487) = −6.88, p < .001.

Regarding nominalizations (lower left-hand graph in Figure 3), the picture was mixed. In the JCCP articles, there was a comparatively small but significant upward trend (r_s = .05, p < .05). In relative terms, this increase was in fact quite substantial, with the frequency of nominalizations increasing by 57.6%, t(1223) = 3.74, p < .001, between the first and the last 15 years under study. In the CCR articles, by contrast, the upward trend did not reach statistical significance (r_s = .07, p = .07). A closer look at the trend reveals that the higher mean in the time period from 1985 to 1999 (.15) was not significantly different from that of the earlier years 1970-1984 (.11), t(383) = 1.88, p = .06, nor from that of the later period 2000-2014 (.12), t(438) = 1.10, p = .27. Thus, the apparent trend reversal regarding nominalizations in CCR that is visible in Figure 3 was not statistically significant.

In line with Billig’s hypothesis, the percentage of articles mentioning “more research is needed” (lower right-hand graph in Figure 3) increased significantly in both CCR (r_s = .10, p < .001) and JCCP articles (r_s = .12, p < .001). When we compared the averages of the 1970-1984 period to those of the 2000-2014 period, the number of articles mentioning “more research is needed” rose by a factor of 5 in both the JCCP articles (1.7% to 8.6%), t(1223) = 5.08, p < .001, and the CCR articles (0.5% to 2.6%), t(487) = 1.83, p = .07.

Discussion

Summary of Findings

Expression of positive and negative emotions

Between 1970 and 2014, the expressions of positive emotions almost doubled across all of the different scales in both abstracts and text bodies of JCCP and CCR articles. We also found a dramatic increase in the use of the top four positive words “robust,” “novel,” “innovative,” and “unprecedented” in the text bodies. Nevertheless, the increase is substantially smaller than the one that was reported in Vinkers et al. (2015). Even though there seems to be a continuing trend toward more positive framing in academic language, this development seems less pronounced in the field of cross-cultural psychology than in medicine.

Notably, we found an increase in expressions of positive emotions not only in the abstracts, but in the text bodies as well. In fact, the increase is even more substantial in the text bodies than in the abstracts. This finding directly contradicts the idea that more marketing-oriented language is only a superficial phenomenon, affecting only the surface of academic writing in the form of catchy titles and advertising language in abstracts. Instead, even the deeper substance, the text bodies (i.e., the main document excluding tables, figures, references, and appendices) are also affected.

Regarding the expression of negative emotions, overall, we found inconsistent results. Whenever monotonic effects reached statistical significance, the effects were rather small. Overall, our findings do not show a consistent increase in expressions of negative emotions.

Frequency of papers mentioning marginally significant effects

The number of studies that reported marginally significant statistical effects almost tripled. These findings resemble very closely the results of Pritschet et al. (2016).

Indicators of technical jargon

We found an increase in words with six or more letters for the JCCP as well as for the CCR articles and abstracts. The increase was in the magnitude of 4% to 11%. Fittingly, the frequency of common verbs decreased in both journals at approximately the same pace. These findings are in line with Billig’s (2011, 2013) criticism of current developments in academic language. We also found a relatively modest increase in the frequency of nominalizations for the JCCP articles, which is compatible with Billig’s observation of increasing pressure on scientific subcommunities to set themselves apart from other communities through the use of new and lofty technical terms, which often serve to reify processes or behavior patterns. It should be noted, however, that the increase we found seems to be dominated by the term “acculturation", which is the most frequent term from our list of nominalizations in the text corpus. John Berry’s acculturation theory (Berry, 1990, 1997) emerged in the 1990s as one of the most important theoretical foundations of cross-cultural psychology. Hence, we tend to regard our findings as rather ambiguous when it comes to the question of whether there is an actual increase in the frequency of nominalization apart from seasonal effects. By contrast, the finding that the frequency of the claim that “more research is needed” quintupled over the years is strong and clear.

Similarities and Differences Between JCCP and CCR

First, we looked at words that predicted most reliably whether an article was from the JCCP or the CCR corpus. We found that the JCCP corpus included more unique psychological, quantitative terms, whereas the CCR represents an anthropological, qualitative approach to culture. Despite these thematic differences, the linguistic trends we observed in this study were surprisingly similar in both journals. There were few noteworthy differences regarding changes in the expression of positive and negative emotions between the JCCP and the CCR articles. Whereas for long words and common verbs we found an increase and decrease during the sampling period for both journals, respectively, long words and common verbs seemed to occur more frequently in the CCR articles than in the JCCP articles at the beginning of the observed time period. Over time, however, JCCP overtook CCR in terms of long word frequency. It can be assumed that the more quantitative focus of the JCCP was playing a role here (see Tables 1 and 2). The same may be true for general differences in the reporting of marginally significant statistical findings and for the percentage of articles mentioning that more research is needed: Here as well, the base rates seem to be slightly higher for the JCCP articles across time. We found an increase only for the JCCP articles in the frequency of nominalizations. As discussed in the previous paragraph, this finding may be related to the increasing popularity of certain theoretical models in JCCP articles, such as the acculturation theory.

How Do the Changes in Cross-Cultural Psychology Compare With Other Scientific Disciplines and Subfields of Psychology?

As discussed above, the increase in positive framing is apparently less pronounced in cross-cultural psychology than it is in medicine and related fields (Vinkers et al., 2015). As of the present time, no studies have compared changes in positive framing in different subfields of psychology. So it remains unclear if the developments that we found in the analysis presented here are part of a general trend in the field of psychology.

Regarding marginal significance reporting, Pritschet et al. (2016) did compare different subfields of psychology, finding that marginally significant effects were reported slightly more frequently in social psychological articles than in developmental psychological and cognitive psychological articles. Still, for all three subfields, there was an overall increase in reporting marginal findings, with a peak around the year 2000. We found a similar pattern in the current study, although the general percentage of articles reporting at least one marginally significant effect appeared to be lower in the field of cross-cultural psychology in comparison with the three subdisciplines that were analyzed in Pritschet et al. (2016, see Figure 2, p. 4). In view of the fact that Pritschet and colleagues used a coding approach, whereas we used an automated linguistic approach, the question remains whether these differences in the absolute frequency of articles reporting marginally significant findings are a result of different methodologies, or whether there is indeed a difference between cross-cultural psychology and the other subfields.

Research on an increasing use of technical jargon in psychology has so far primarily used a qualitative method, in the form of an in-depth analysis of a small number of selected articles (e.g., Billig, 2008, 2011, 2013). Hence, it is not possible at the present time to compare the linguistic changes that we found for the field of cross-cultural psychology (an increase in long words, a decrease in common verbs, and—at least for the JCCP articles—a slight increase in nominalizations) with other scientific disciplines or other psychological subdisciplines.

Limitations

Our automated linguistic approach is prone to a number of potential biases and distortions. Double meanings of terms are a common problem in any linguistic investigation of a large corpus of material (e.g., Acerbi et al., 2013). For example, the term “phenomenal” can on one hand be a synonym for “fantastic” and hence be an expression of positive emotion; on the other hand, it can refer to a “phenomenal” investigation in the form of studying concrete occurrences instead of abstract theories. The magnitude of the text corpus we analyzed rendered it impossible to counter-check every single assignment of a word to a given category. Whenever possible, we addressed this issue by means of several indicators of the same phenomenon (e.g., the different scales for expressions of positive affect) as a means of a methodological triangulation (Denzin, 1978).

It is also not possible to rule out the possibility that some cases for which researchers reported marginally significant statistical effects went unnoticed, because these effects were not explicitly described as marginally significant. Such attempts as obfuscating the marginally significant status of statistical findings could limit the validity of our findings.

Whereas we argue that the increase in expressions of positive emotions in these texts does not simply mirror global trends in common language but instead is a symptom of a specific trend in science, the possibility remains that developments within the field of psychology such as the emergence of the positive psychology movement (Seligman & Csikszentmihalyi, 2000) influenced our findings. However, we think that at least the “top four positive words” scale comprising only four words which are very indicative of the increasing use of more advertising and exaggerative language should not be affected by any general trend toward more positivity. At the same time, we cannot rule out the possibility that cross-cultural psychological findings have actually become more “robust,” “novel,” “unprecedented,” and “innovative.”

The question remains to what degree the linguistic changes that we and other researchers have found are related to changes in academic practices such as study design, sampling, data collection, and analysis. One could hypothesize, for example, that the increasing use of language that is indicative of self-marketing should probably correspond with a higher willingness to use questionable research practices (John, Loewenstein, & Prelec, 2012), as long as those practices help to increase a researcher’s visibility and number of publications. Leggett, Thomas, Loetscher, and Nicholls (2013) found indeed an increase in p values slightly below the “magical” threshold of 0.05 for psychological articles published between 1965 and 2005. However, in a recent study, Nuijten, Hartgerink, van Assen, Epskamp, and Wicherts (2015) used the R-package statcheck (Epskamp & Nuijten, 2015) to analyze errors in the reporting of statistical results in articles from eight major psychological journals that were published between 1985 and 2013. Overall, they did not find any increase over time (rather, a slight decrease). Hence, the question as to whether the linguistic changes in academic language correspond as well to changes in questionable research practices cannot be answered conclusively at the present time. For future studies, it might be a promising strategy to combine the linguistic approach described in this paper with algorithms for the detection of questionable research practices, such as statcheck, to test the hypothesis that there may indeed be a language of sloppy science.

With regard to our dictionary for marginally significant statistical effects, we can of course not discern whether the authors reported marginally significant effects themselves or whether they were referring to studies of colleagues. However, we think that actually both cases are indicative of a willingness to display even small and uncertain statistical effects as significant.

LIWC’s list of common verbs was not designed to capture typical verbs in scientific publications. Still, we find it noteworthy that even for such a rather arbitrary collection of verbs, a relatively robust, significant, and steady decrease can be found within the time period of our analysis. Together with our findings on the increasing frequency of long and presumably complicated words, we believe that—with all due carefulness—these results can be interpreted to be consistent with Billig’s qualitative analysis of changes in academic language.

Of course, it would be interesting to additionally investigate changes in the prevalence of passive voice constructions. Here as well, Billig predicted an increase parallel to the increase of nominalization. At the present time, we could not find any software tool which could assess passive voice constructions in such a large text corpus reliably enough.

Implications and Conclusion

The current study shows the prevalence of positive framing, the exaggeration of small effects, and the use of overly complicated language in research reports in the field of cross-cultural psychology. All of these indicators might, however, have a common cause, namely the rise of academic capitalism (Münch, 2014; Slaughter & Rhoades, 2004). The academic writing style today increasingly includes self-marketing strategies as a consequence of increasing publication pressure and the author’s need to become visible as a researcher.

Michael Billig (2013) has suggested a number of recommendations for researchers to improve the linguistic quality of academic publications. Interestingly, his recommendations stand in the tradition of George Orwell’s (1946/2013) critique of political language: use simple language whenever possible, reduce the number of passive sentences (this is in line with the American Psychological Association [APA] style guide), preferably use verb phrases over noun phrases, re-personalize psychological texts by writing more about persons and less about artificial processes and phenomena by utilizing big words, and avoid being overly attached to technical terms to avoid adopting a sales representative’s attitude in promoting your pet concepts over others. He also recommended not to take his recommendations too seriously and to ignore them whenever it seems appropriate.

It should be noted that Orwell’s original suggestions were referring to the language employed in authoritarian regimes such as the Soviet Union. In authoritarian language, linguistically deleting the actors and replacing common language expressions with technical terms serves the function of immunizing the ruling regime against criticism, suppressing alternative discourse in society, and bringing society into line. It is of course a somewhat frightening idea that changes in academic language resemble changes that in the realm of politics go along with a loss of personal freedom and societal diversity. We do not want to say that this is what has been happening in academia over the last decades, but we do wish to state that the time is ripe for increased awareness of the language of science, not only in cross-cultural psychology, but in other academic fields as well.

We believe that at the moment, psychology is entering a stage of increasing self-reflection and self-criticism (e.g., Pinker, 2014; Schmidt & Oh, 2016; Stroebe, 2016). One aspect that so far has been widely neglected in any discussion that focuses on reproducibility and statistical methods, but which nevertheless may be closely related, is the language of academic publications. We hope that our contribution will help to stimulate this debate by shedding light on linguistic changes in academic publications over the last several decades. We are optimistic that the act of debating these issues in itself will help psychology to move forward and to master the challenges of science in the 21st century.

Footnotes

Appendix

Table A3.

Unpaired t Tests Comparing the Mean of the First 15 Years (1970-1984) With the Mean of the Last 15 Years Under Study (2000-2014) in CCR.

	1970-1984			2000-2014			Difference in means	t statistic
	n	M	SD	n	M	SD	Difference in means	t statistic
Abstracts
Positive emotions (LIWC)	157	1.689	1.708	271	2.881	2.273	1.192***	−5.7
25 positive words (Vinkers et al., 2015)	157	0.000	0.000	271	0.040	0.187	0.040**	−2.68
Negative emotions (LIWC)	157	1.374	1.812	271	1.329	1.970	−0.045	−0.23
25 negative words (Vinkers et al., 2015)	157	0.020	0.141	271	0.012	0.089	−0.008	−0.67
Full texts
Positive emotions (LIWC)	217	1.548	0.791	272	2.159	1.079	0.612***	−6.98
25 positive words (Vinkers et al., 2015)	217	0.023	0.039	272	0.038	0.051	0.015***	−3.52
Top four positive words (Vinkers et al., 2015)	217	0.003	0.013	272	0.012	0.059	0.009*	−2.14
Negative emotions (LIWC)	217	1.243	0.957	272	1.246	1.061	0.003	−0.03
25 negative words (Vinkers et al., 2015)	217	0.033	0.073	272	0.019	0.027	−0.014**	−2.85
Marginal effects	217	3.687	18.887	272	14.706	35.482	11.019***	−4.13
“More research is needed”	217	0.461	6.788	272	2.574	15.864	2.113	−1.83
Words with six or more letters	217	30.281	3.683	272	31.594	3.414	1.312***	−4.08
Common verbs	217	7.951	1.996	272	6.906	1.354	−1.045***	−6.88
Nominalizations	217	0.108	0.184	272	0.123	0.245	0.015	−0.75

Note. CCR = Cross-Cultural Research; LIWC = Linguistic Inquiry and Word Count.

p < .05. **p < .01. ***p < .001.

Acknowledgements

The authors thank Manfred Knobloch from the Leibniz-Institut für Wissensmedien (IWM) greatly for his invaluable help in preparing the data for the analysis. They are also grateful to Alan Maloney and SAGE Publications Ltd. for granting us access to the full texts of the Journal of Cross-Cultural Psychology (JCCP) and Cross-Cultural Research (CCR) articles that were analyzed in this study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Acerbi

Lampos

Bentley

R. A.

(2013, October 6-9). Robustness of emotion extraction from 20th century English books. 2013 IEEE International Conference on Big Data, Santa Clara, CA.

Allik

(2013). Bibliometric analysis of the Journal of Cross-Cultural Psychology during the first ten years of the new millennium. Journal of Cross-Cultural Psychology, 44, 657-667.

Anthony

(2005, July 10-13). AntConc: Design and development of a freeware corpus analysis toolkit for the technical writing classroom. Proceedings of the International Professional Communication Conference IPCC, Limerick, Ireland.

Berry

J. W.

(1990). Psychology of acculturation. In Berman

J. J.

(Ed.), Nebraska symposium on motivation, 1989: Cross-cultural perspectives. Current theory and research in motivation (Vol. 37, pp. 201-234). Lincoln: University of Nebraska Press.

Berry

J. W.

(1997). Immigration, acculturation, and adaptation. Applied Psychology, 46, 5-34.

Best

D. L.

Everett

B. S.

(2010). The most recent years: The Journal of Cross-Cultural Psychology, 2004-2009. Journal of Cross-Cultural Psychology, 41, 329-335.

Biber

Conrad

Cortes

(2004). If you look at . . .: Lexical bundles in university teaching and textbooks. Applied Linguistics, 25, 371-405.

Biber

Gray

(2010). Challenging stereotypes about academic writing: Complexity, elaboration, explicitness. Journal of English for Academic Purposes, 9, 2-20.

Billig

(2008). The language of critical discourse analysis: The case of nominalization. Discourse & Society, 19, 783-800.

10.

Billig

(2011). Writing social psychology: Fictional things and unpopulated texts. British Journal of Social Psychology, 50, 4-20.

11.

Billig

(2013). Learn to write badly: How to succeed in the social sciences. Cambridge, UK: Cambridge University Press.

12.

Boucher

Osgood

C. E.

(1969). The pollyanna hypothesis. Journal of Verbal Learning and Verbal Behavior, 8(1), 1-8.

13.

Brouwers

S. A.

Van Hemert

D. A.

Breugelmans

S. M.

Van de Vijver

F. J. R.

(2004). A historical analysis of empirical studies published in the Journal of Cross-Cultural Psychology 1970-2004. Journal of Cross-Cultural Psychology, 35, 251-262.

14.

Castells

(2010). End of millennium: The information age: Economy, society, and culture. Oxford, UK: Wiley-Blackwell.

15.

Denzin

N. K.

(1978). Sociological methods: A sourcebook. New York, NY: McGraw-Hill Companies.

16.

Dodds

P. S.

Danforth

C. M.

(2010). Measuring the happiness of large-scale written expression: Songs, blogs, and presidents. Journal of Happiness Studies, 11, 441-456.

17.

Doyle

Cuthill

(2015). Does “get visible or vanish” herald the end of “publish or perish”? Higher Education Research & Development, 34, 671-674.

18.

Epskamp

Nuijten

M. B.

(2015). Statcheck: Extract statistics from articles and recompute p values (R package version 1.0. 1.). Retrieved from https://cran.r-project.org/web/packages/statcheck/index.html

19.

Flesch

(1948). A new readability yardstick. Journal of Applied Psychology, 32, 221-233.

20.

Garcia

Garas

Schweitzer

(2012). Positive words carry less information than negative words. EPJ Data Science, 1(1), Article 3.

21.

Gelman

Carlin

J. B.

Stern

H. S.

Rubin

D. B.

(2014). Bayesian data analysis (Vol. 2). Boca Raton, FL: Chapman & Hall.

22.

Gigerenzer

(2004). Mindless statistics. The Journal of Socio-Economics, 33, 524-432.

23.

Gutierrez

R. G.

Linhart

J. M.

Pitblado

J. S.

(2003). From the help desk: Local polynomial regression and Stata plugins. Stata Journal, 3, 412-419.

24.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532.

25.

Kilgarriff

(2001). Comparing corpora. International Journal of Corpus Linguistics, 6, 97-133.

26.

Leggett

N. C.

Thomas

N. A.

Loetscher

Nicholls

M. E.

(2013). The life of p: “Just significant” results are on the rise. The Quarterly Journal of Experimental Psychology, 66, 2303-2309.

27.

Moore

Neylon

Eve

M. P.

O’Donnell

D. P.

Pattinson

(2017). Excellence R us: University research and the fetishisation of excellence. Palgrave Communications 3, Article number: 16105. Retrieved from https://www-nature-com-s.web.bisu.edu.cn/articles/palcomms2016105

28.

Münch

(2014). Academic capitalism: Universities in the global struggle for excellence. New York, NY: Routledge.

29.

Nuijten

M. B.

Hartgerink

C. H.

van Assen

M. A.

Epskamp

Wicherts

J. M.

(2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48, 1205-1226.

30.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

31.

Orwell

(2013). Politics and the English language. London, England: Penguin. (Original work published 1946)

32.

Pashler

Wagenmakers

E. J.

(2012). Editors’ introduction to the special section on replicability in psychological science a crisis of confidence? Perspectives on Psychological Science, 7, 528-530.

33.

Pechenick

E. A.

Danforth

C. M.

Dodds

P. S.

(2015). Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution. PLoS ONE, 10(10), e0137041.

34.

Pennebaker

J. W.

(1993). Putting stress into words: Health, linguistic, and therapeutic implications. Behaviour Research and Therapy, 31(6), 539-548.

35.

Pennebaker

J. W.

Boyd

R. L.

Jordan

Blackburn

(2015). The development and psychometric properties of LIWC2015. Austin, Texas: UT Faculty/Researcher Works.

36.

Pinker

(2014, September 26). Why academics stink at writing. Chronicle of Higher Education. Retrieved from http://www.chronicle.com/article/Why-Academics-Writing-Stinks/148989

37.

Pritschet

Powell

Horne

(2016). Marginally significant effects as evidence for hypotheses: Changing attitudes over four decades. Psychological Science, 277, 1036-1042. doi:10.1177/0956797616645672

38.

Sarewitz

(2016). The pressure to publish pushes down quality. Nature, 533(7602), 147.

39.

Schmidt

F. L.

I. S.

(2016). The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4, 32-37.

40.

Schneickert

Lenger

(2016). Studentische Arbeitskraftunternehmer*innen: Projektbasierter Kapitalismus im wissenschaftlichen Feld [Student assistants as entrepreneurs: Project-based capitalism in the academic field]. In Vester-Lange

Sander

(Eds.), Soziale Ungleichheiten, Milieus und Habitus im Hochschulstudium [Social inequalities, milieus, and habitus in higher education] (pp. 265-285). Weinheim, Germany: Beltz Juventa.

41.

Seligman

M. E.

Csikszentmihalyi

(2000). Positive psychology: An introduction. The American Psychologist, 55, 5-14.

42.

Slaughter

Rhoades

(2004). Academic capitalism and the new economy: Markets, state, and higher education. Baltimore, MD: Johns Hopkins University Press.

43.

Smith

P. B.

Bond

M. H.

Kagitcibasi

(2006). Understanding social psychology across cultures: Living and working in a changing world. Thousand Oaks, CA: Sage.

44.

Stroebe

(2016). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134-144. doi:10.1016/j.jesp.2015.09.017

45.

van de Vijver

F. J. R.

Lonner

W. J.

(1995). A bibliometric analysis of the Journal of Cross-Cultural Psychology. Journal of Cross-Cultural Psychology, 26, 591-602.

46.

Vinkers

C. H.

Tijdink

J. K.

Otte

W. M.

(2015). Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: Retrospective analysis. British Medical Journal, 351, Article h6467.

47.

Wittgenstein

(1922). Tractatus logico-philosophicus. Side-by-side edition, version 0,45 (June 2017). Retrieved from https://people.umass.edu/klement/tlp/tlp.pdf