Instructor Evaluation of Business Student Writing: Does Language Play a Role?

Abstract

Because few studies of disciplinary business writing have examined whether language features play a role in instructor assessment of student writing, this study explored the relationship between student language use and instructor essay scores. Undergraduate business students wrote a case study critique as part of their final exam, and their critiques were evaluated by their instructors for theory integration and essay structure. Student language use was analyzed in terms of error rate, lexical sophistication, lexical diversity, and phrasal complexity. Whereas lexical sophistication positively predicted instructor scores, error rate was a negative predictor of their assessment of business student writing.

Keywords

case study critique business student writing errors phrasal complexity lexical sophistication

The importance of both oral and written communication skills for business students is widely recognized by university faculty, employers, and accrediting bodies (e.g.,AACSB International, 2020). Recent faculty and employer responses to the National Association of Colleges and Employers job outlook survey indicated that communication skills were ranked within the top four competencies for another year (National Association of Colleges and Employers, 2019), and numerous survey studies have highlighted the value attached to communication skills by various stakeholders (Blanthorne et al., 2005;Conrad & Newberry, 2011;Elrod et al., 2014;Schartel Dunn & Lane, 2019;Wardrope, 2002;Wu & Kendall, 2006). Reflecting the importance of these skills, business programs have adopted a wide variety of instructional approaches to help students acquire written communication skills, ranging from standalone classes for business students (Moshiri & Cardon, 2014) to writing across the business curriculum models that integrate target skills into core business courses (e.g.,Hutchins, 2015). Nevertheless, survey studies have reported that employers find the communication skills of new hires lacking (Ghannadian, 2013) and that supervisors regard the writing and grammar skills of interns as inadequate (Schartel Dunn & Lane, 2019). In light of the value attached to communication skills in business, it is important to examine how instructors assess disciplinary writing and whether they orient to language use when evaluating student writing.

Due to the importance of written communication skills, business programs teach students how to write in professional genres for the workplace and disciplinary writing tasks to demonstrate mastery of business content (Zhu, 2004a). Professional genres such as memos, emails, credit reports, letters, executive summaries, proposals, and case studies (Chan, 2019;Fraser et al., 2005;Hutchins, 2015;Rogers, 1994;Sigmar & Hynes, 2012) help prepare students for workplace written communication while also requiring them to demonstrate curricular knowledge (Zhu, 2004b). In contrast, the purpose of disciplinary writing tasks is for students to display and solidify their knowledge of a discipline, which is common practice across academic disciplines (Gimenez, 2017;Nesi & Gardner, 2006;Nicolas & Annous, 2013;Smart et al., 2013). In business programs, case-based teaching is widely used to help students demonstrate and develop an understanding of professional practice or knowledge by analyzing a single case (Jackson, 2004). As described byNathan (2013), case studies can take a variety of different forms (i.e., critique, report, analysis, and presentation). In the context investigated here, students write case study critiques in which they use business theories and principles to analyze and evaluate a case.

When assessing disciplinary writing such as case study critiques, instructors expect students to demonstrate understanding of core business principles and theories; therefore, instructor evaluation of student writing concentrates on students’ understanding of content. Similar to academic writing in other disciplines, business students are held accountable for the content of their texts, that is, text-responsible writing (Leki & Carson, 1997;Zhu, 2004b). In other words, students engage in knowledge display by producing documents that show an expert how well they have understood concepts, theories, and principles through writing (Gimenez, 2017;Smart et al., 2013). Interviews with business faculty have shown that they view their primary role as teaching content and technical skills (not writing) and their feedback concentrates on content and correct understanding of course information (Zhu, 2004a). In a previous study about the academic success of international students enrolled in the business program examined here (Neumann et al., 2019), instructors reported that understanding of business theories and the application of that knowledge were important for student success. The interview data further revealed that they evaluated student writing in terms of theoretical content and business principles but did not prioritize correcting language use or giving language feedback. In sum, disciplinary business writing emphasizes students’ mastery of content to a greater extent than their language use.

Written Language Features

Errors

Despite their emphasis on the application of theoretical knowledge, instructors may also be sensitive to language when evaluating student writing. In recognition of the importance of effective communication skills for accreditation agencies and future employers, instructors may pay attention to language in subtle ways when evaluating student texts even if they view their primary role as facilitators of disciplinary knowledge. Due to their salience and the importance of avoiding them in workplace communication, errors have been widely researched in terms of whether they are noticed and which ones are more irritating or “bothersome” (e.g.,Boettger & Emory Moore, 2018;Brandenburg, 2015;Martin-Lacroux & Lacroux, 2017). When assessing professional writing tasks such as memos for assessment purposes, errors are often included in the rubrics and evaluation criteria (Fraser et al., 2005;Rogers, 1994). Less is known, however, about whether errors contribute to the assessment of disciplinary writing when instructors give primary emphasis to content.

Phrasal Complexity

Whereas errors are a salient language feature that might play a role in the assessment of business student writing, language features associated with the genre of academic writing may also influence instructor evaluations. Using a quantitative, corpus-based technique called multidimensional analysis, which identifies patterns of linguistic variation, Biber and colleagues have identified the language features that characterize academic writing (such as journal articles, textbooks, and course materials) as opposed to spoken genres and other written genres (such as fiction, letters, and press reports) in British and American English (Biber, 1988;Biber et al., 1999) as well as American university discourse (Biber et al., 2002). These large-scale corpus analyses identified several language features that are characteristic of academic writing. For example, whereas spoken language contains clausal coordination and subordination, written academic discourse is characterized by phrasal complexity. More specifically, the following phrasal features are more prevalent in journal articles than conversation: attributive adjectives (extrinsic factors), nouns as prenominal modifiers (leadership style), and postnominal prepositional phrases (success of the company;Biber et al., 2011). The authors explained the clustering of these features as reflecting information density because they allow writers to communicate disciplinary knowledge efficiently.

Corpus linguistic analysis of student disciplinary writing has shown that business writing has informational density features including attributive adjectives and prenominal nouns (Gardner et al., 2019;Staples et al., 2018). When these studies compared student texts from different degree programs, texts written by graduate students contained more information density features than those written by undergraduate students. Furthermore, the comparison of academic texts written by undergraduate students in different years of study also showed an increase in information density features, specifically attributive adjectives and prenominal nouns (Staples et al., 2016), which provides evidence that phrasal complexity may increase as students develop their academic writing skills. In sum, the corpus linguistic studies to date have confirmed that academic writing is associated with phrasal complexity and that students’ use of phrasal features increases over time. Due to their prevalence in academic writing and their function to express disciplinary content succinctly, phrasal complexity may positively influence instructor evaluation of business student writing.

Lexical Diversity and Sophistication

Turning to additional language features that could potentially influence instructors’ assessment of business student writing despite their overt focus on content, lexical diversity has been linked with positive assessment of student writing. Lexical diversity refers to the number of different words that a writer uses, which is measured by considering types (unique words) and tokens (occurrence of the same word). The simplest measure of lexical diversity is the type-token ratio, which is calculated by dividing the number of types by the number of tokens. However, this ratio tends to decrease as text length increases due to the repetition of words in longer texts. As a result, more sophisticated lexical diversity measures have been developed to counteract the effect of text length. For example, the D index developed byMalvern et al. (2004)models curves of type-token ratios against tokens to find the best fitting curve for a text. The higher the D value, the greater a text’s lexical diversity. Comparative studies of argumentative essays written by secondary (Grades 9 and 11) and first-year university students (Crossley et al., 2011) showed that mean lexical diversity scores increased over time from approximately 55 for 9th graders to 72 for 11th graders and 86 for first-year university students.Durán et al. (2004)reported that the lexical diversity values for academic texts ranged from approximately 70 to 110, with a mean of 90.

Whereas lexical diversity captures vocabulary range, lexical sophistication is evaluated in relation to the frequency of the words used in the text, with an infrequent lexical item considered more sophisticated than a commonly used word. Frequency is generally determined through reference to established frequency bands or presence on academic word lists. In addition to frequency benchmarks, mean word length is also taken as a proxy for lexical sophistication following the logic that sophisticated words are generally longer than more frequent and common words. Using longitudinal data from first- and third-year university students,Haswell (2000)reported that the frequency of long words (more than nine letters) increased over time. More recently, the corpus research that examined phrasal complexity described previously (Gardner et al., 2019;Staples et al., 2018) reported that word length was associated with the information density dimension of business writing. Due to its association with informational density, lexical sophistication measured as word length may also play a role in the assessment of disciplinary business writing.

The Current Study

To summarize, although business student writing is typically assessed by evaluating how well they have understood, applied, or evaluated disciplinary content, instructors may be sensitive to language when assessing student texts. More specifically, instructors may orient to errors due to their salience and the importance of avoiding them in professional workplace communication. In addition, instructors may be sensitive to the informational density features associated with academic writing generally, which include phrasal complexity, lexical diversity, and lexical sophistication. Therefore, the current study explores whether these language features are related to instructor evaluation of business student writing, specifically case study critiques. The research question is as follows: Which language features (errors, phrasal complexity, lexical diversity, or lexical sophistication) predict instructor evaluation of business students’ case study critiques?

Method

Focusing on student case study critiques written in a business communication course, the current study addressed the research question by analyzing whether there was a relationship between the students’ language use and the scores they received from their instructors. In the following sections, we first describe the students and the instructional focus of their business communication course. We next explain the content, procedure, and assessment criteria for the case study critique that students wrote as part of their final examination in the course. Finally, we provide details about the analytic tools and steps taken to identify the occurrence of the target language features in their texts.

Students and Instructional Context

The participants were 77 undergraduate students (46 men, 31 women) enrolled in a disciplinary business course at an English-medium university in Montreal, Canada. They ranged in age from 17 to 32 years with a mean age of 20.6 years (SD= 2.7). Most of the participants were studying business degrees in finance (26%), accounting (25%), international business (16%), marketing (13%), and business technology management (10%), while the remaining students (10%) were enrolled in related degree programs in the social sciences, such as economics or communication studies. Reflecting the Montreal linguistic environment, the students predominantly spoke English (31%) or French (31%) as their first languages (L1s). Besides English and French, they reported speaking 14 other L1s, with Arabic (8%), Mandarin (6%), and Spanish (6%) most frequently mentioned.

The students were taking a required communication course (Contemporary Business Thinking) in the core business curriculum that had multiple class sections taught by different instructors. The goals of the course were to familiarize students with foundational business theories and foster critical analysis of core business texts and case studies. The course assessments included participation and assignments (10%), a team project (30%), and two examinations: midterm (20%) and final (40%). The data analyzed here came from the final examination, which consisted of four case study readings with seven prompts to be handwritten in three hours. The prompts instructed students to respond to the cases using specific theoretical or conceptual frameworks that had been discussed in the course lectures and readings, such as economies of scale, core ideologies, and organizational culture. The first three prompts elicited short answers (half page to one page), but the fourth critique elicited a longer response (two pages) that was scored out of 20 points. Students were required to fit their critique in the amount of space allotted for each prompt. Due to its greater length, we analyzed the final case study critique, which asked students to draw on three leadership frameworks to evaluate how Jeff Bezos’ leadership style impacts Amazon’s success. The prompt instructed students to write a clear claim, provide good evidence, anticipate and argue potential objections, and include a conclusion.

Case Study Critique and Scoring

The participants wrote a case study critique as part of their final exam, which was a required assessment task in their business course. At the beginning of the semester, research assistants visited each class to describe the research project and request written consent to access the students’ exams after their instructors evaluated the exams and submitted course grades. Prior to sharing the exams with the researchers, the instructors assessed the case study critiques following the uniform scoring rubrics provided by the course coordinator. For the Amazon case study critique, the rubric consisted of two dimensions: theory integration (15 points) and essay structure (5 points). To assess theory integration, instructors considered how well students had demonstrated understanding of three leadership frameworks specified in the prompt along with the quality of evidence used to support each theory (5 points per theory). To evaluate essay structure, instructors considered discourse features such as the presence of a concise claim (1 point), provision of evidence (2 points), rebuttals of counter claims (1 point), and “persuasive and vivid writing with proper structure of intro, paragraphs and conclusion” (1 point). The total essay score was 20 points, and the essay score was taken as the outcome variable.

Analysis of Language Features

To explore whether language features predicted essay scores despite the absence of explicit language criteria in the evaluation rubric, we considered four language variables associated with business writing in previous studies. First, we included error rate (number of errors/total words) because of the perceived importance of errors by both academic and business practitioners (e.g.,Boettger & Emory Moore, 2018;Martin-Lacroux & Lacroux, 2017). The second researcher coded the essays for errors usingPolio and Shea’s (2014)error list which was modified fromPolio (1997). Their framework consists of 25 different error types that include sentence fragments, run-on sentences, problems with relative clause formation, tense/aspect, incorrect word formation, and preposition errors. The number of errors was determined by the minimal number of corrections necessary to make a phrase or clause error-free. If multiple errors occurred on a single word, such as numerous problems with tense, aspect, mood, or verb formation, it was counted as one error. Errors in word choice were included only when the word or expression distorted the meaning. Capitalization errors and minor punctuation errors, such as confusion between parentheses and brackets or between single and double quotation marks, were ignored because they rarely occurred in the essays and have been shown to be less “bothersome” than other types of errors (Boettger & Emory Moore, 2018).Table 1provides examples of the error types coded in the students’ essays.

Table 1.

Examples of Error Coding.

Error type	Example
Sentence fragment	First of all, because he is not able to motivate his employees effectively.
Run-on sentence	A leader aspires, motivates and works with while on the other side a boss pushes discourages and under appreciates, therefore in a case like Amazon, having a cruel managerial way will eventually lead to the failure of the company.
Relative clause	Then there is expert power for peoplewhichhave expertise in a field, for example doctors, lawyers.
Verb tense	I believe that Jeff Bezos’ leadership style ultimatelyleadto a decline in Amazon’s success.
Subject-verb agreement	Herzberg states that true motivationcomefrom intrinsic job satisfaction.
Word formation	This not only encouragesunmotivationbut also gives employeesun-satisfaction.
Preposition choice	The study was focusedinletting the employees be motivated and inspired rather than being controlled.

A research assistant was trained to carry out error coding on a subset of the data (20%). Interrater reliability was assessed using two-way mixed interclass correlation coefficient, which was .76. The coefficient reached the acceptable benchmark for reliability for language research (i.e., .70 to .80;Larson-Hall, 2010). After all errors were identified and summed, a proportion score was obtained by dividing the number of errors by the total words in the text.

Second, because of its association with informational density in academic writing, phrasal complexity was included in the analysis. It was operationalized as complex nominals per clause (Lu, 2010), which includes nouns with prenominal modifiers (adjectives, nouns, and possessive nouns), nouns with postnominal modifiers (prepositional phrases, relative clauses, participles and appositives), nominal clauses, and gerund and infinitive subjects.Table 2provides examples of the different types of complex nominals measured for phrasal complexity. To obtain the phrasal complexity score, typed versions of the students’ handwritten texts were submitted toLu’s (2010)syntactic complexity analyzer, which is available online (https://aihaiyang.com/software/l2sca/).

Table 2.

Phrasal Complexity Measures.

Measure	Definition	Examples
Complex nominals/clause	The number of nominals that have (i) nouns plus adjective, possessive, prepositional phrase, relative clause, participle, or appositive, (ii) nominal clauses, and/or (iii) gerunds and infinitives as subjects divided by the number of clauses.	The complex decision-making style of Jeff Bezos(i) demotivates some employees. For example,packing products in warehouses(iii) is a demotivating taskfor the employees in the company(i). Herzberg statesthat true motivation comes from intrinsic job satisfaction(ii).

Finally, both lexical diversity (D) and lexical sophistication (average word length) were considered as having a potentially positive relationship with essay scores. Examples of lexical diversity and sophistication are provided inTable 3. To obtain both scores, the typed versions were submitted online to Coh-Metrix (http://cohmetrix.com) after omitting article titles and quotations.

Table 3.

Measures of Lexical Diversity and Lexical Sophistication.

Feature	Measure	Explanation	Example
Lexical diversity	D	Taking into consideration text length, a wider range of different vocabulary words are used	Rather than use one word repeatedly (e.g.,leader), a writer uses synonyms (CEO, boss, manager, employer, head)
Lexical sophistication	Mean word length	Sophisticated words are generally longer than more frequent and common words	Implementation, legitimate, anddevelopmentare more sophisticated thanusage, legal, andgrowth

Results

The research question asked which language features predict instructors’ assessment of business student writing, specifically case study critiques. The students’ case study critiques ranged in length from 234 to 703 words, with a mean length of 453 words (SD= 97), with only two students writing less than 300 words. Because our focus was on specific language features associated with academic writing, we did not include text length as a potential predictor variable. In other words, we were interested in whether the type of language used was related to instructor evaluation, not the quantity of language produced. The descriptive statistics for the outcome variable (instructor score) and predictor variables (error rate, phrasal complexity, lexical diversity, and lexical sophistication) are provided inTable 4.

Table 4.

Descriptive Statistics for Outcome and Predictor Variables.

Variable	M	SD
Instructor score (out of 20)	15.20	2.40
Error rate: Errors/words	0.06	0.03
Phrasal complexity: Complex nominals/clause	1.19	0.24
Lexical diversity: D	103.56	18.30
Lexical sophistication: Mean word length	4.98	0.21

Prior to entering the predictor variables into a regression model, Pearson correlation coefficients were obtained to determine the extent to which each one was related to instructor scores. The goal of this preliminary analysis was to identify the variables that had at least a small relationship with instructor scores for inclusion in the regression model, with the relationship evaluated according to benchmarks for applied linguistics research that define a small relationship as being at least ±.25 (Plonsky & Oswald, 2014). As shown inTable 5, two language features reached the benchmark: error rate and lexical sophistication. Whereas lexical sophistication had a positive relationship with instructor scores, error rate had a negative relationship (i.e., as error rates increased, instructor scores decreased). Both lexical diversity and phrasal complexity had marginal (r= .02) relationships with instructor scores, so they were excluded from the regression model.

Table 5.

Correlation Coefficients for Instructor Scores and Predictor Variables.

Predictor variables	r	p
Error rate	−.43	.001
Phrasal complexity	.02	.841
Lexical diversity	.02	.853
Lexical sophistication	.28	.013

Because research has examined both errors (Beason, 2001;Boettger & Emory Moore, 2018;Martin-Lacroux & Lacroux, 2017;Wolfe et al., 2016) and lexical sophistication (Gardner et al., 2019;Staples et al., 2018) in business writing previously, they were entered into the model in the same step. The regression model was significant,F(2, 76) = 11.43,p< .001, accounting for 22% of the total variance (adjustedR ²) in instructor scores. To ensure that the model was not affected by a strong correlation between error rate and lexical sophistication, we confirmed that the tolerance value was above .20 (= .99) and the variance inflation factor value was not above 10 (= 1.02). To confirm that the model was not affected by outliers, we examined the standardized residuals. Only 5% of the cases had standardized residuals greater than ±2, which suggests that bias in the model was not a concern. Also, all cases had Cook’s distance values below one, which indicates there was an absence of cases with undue influence on the model.

Having confirmed model fit, we then examined the predictor variables. As shown inTable 6, both error rate and lexical sophistication were significant predictors of instructor scores. The positive standardized beta value (.24) for lexical sophistication indicates that increases in the use of longer words is associated with higher instructor scores when error rate is held constant. In contrast, the negative value associated with error rate (−.40) indicates that increases in error rate are predictive of lower instructor scores when lexical sophistication is held constant.

Table 6.

Summary of Predictor Variables for Regression Model With Both Blocks.

Predictors	B	SEB	Exp (B)	95% CI		t	p
Predictors	B	SEB	Exp (B)	Lower limit	Upper limit	t	p
Error rate	−39.37	10.11	−.40	−59.51	−19.22	−3.89	.001
Lexical sophistication	2.71	1.17	.24	.37	5.05	2.31	.024
Constant	−3.87	5.95

Note. SE= standard error; CI = confidence interval.

As mentioned in the Participants section, these students represented the linguistic diversity of Montreal in terms of being L1 speakers of English (31%), French (31%), and other languages (38%). Although our primary interest is in the relationship between language features and instructor scores, we recognize that the students’ language background may be a mediating variable. To explore the potential role of L1 background, we carried out a post hoc analysis of the relationship between L1 and instructor scores, focusing specifically on the case study critiques that received high and low scores. Using the mean score (15.20) to classify students into high and low performance groups, we first removed all students who scored near the mean (14.5 to 16). Students who scored at least 16.5 were classified as high-scorers (n= 22) while students who scored a maximum of 14 were classified as low scorers (n= 23). As shown inTable 7, there was no clear relationship between exam performance and L1 background. In each of the three L1 backgrounds, 42% to 57% of the students were high scorers. A chi-square test indicated that there was no significant relationship between L1 background and essay scores: χ²(4, 55) = 1.29,p= .863. Thus, there is no evidence that L1 English speakers clustered in the high-score category while students from other L1 backgrounds were in the low-score category.

Table 7.

Essay Scores by L1 Background.

	English	French	Other
High scorers (n= 22)	8	5	9
Low scorers (n= 23)	6	7	10

Because differences in error rates might be expected based on L1 background, we also compared the error rate among L1 groups. The mean error rate was similar across L1 groups: EnglishM= 0.05,SD= 0.02; FrenchM= 0.06,SD= 0.03; Other L1M= 0.06,SD= 0.02. In light of the null findings for case study scores and error rates, we did not carry out additional post hoc comparisons for the other language features. In sum, the post hoc analyses confirm prior comparative studies that found similarities in the performance of students from diverse language backgrounds on disciplinary writing tasks (Weigle & Friginal, 2015).

Discussion

To summarize the main findings, instructor scores for business student writing were predicted by error rate and lexical sophistication. Whereas errors predicted lower instructor scores, lexical sophistication predicted higher scores. Even though the evaluation criteria emphasized business content criteria, these two language features accounted for 22% of the variance in instructor scores. The post hoc analyses indicated that there were no differences in instructor scores or errors based on L1 background. Our findings confirm prior studies that found a positive relationship between word length as an informational density feature and academic writing (Gardner et al., 2019;Staples et al., 2018). Whereas prior longitudinal research found that lexical sophistication increased over time (Haswell, 2000), our findings revealed that it also predicts instructor scores. Finally, the error findings confirm prior studies that have documented negative reactions to errors in business writing (Beason, 2001;Boettger & Emory Moore, 2018;Martin-Lacroux & Lacroux, 2017;Wolfe et al., 2016).

Turning to the variables that did not predict case study exam scores, the null finding for lexical diversity disconfirms prior research that has shown that it captures variation in writing performance (Crossley et al., 2011;McNamara et al., 2010). One possible reason for the divergent findings is that our sample did not include sufficient variation to detect the potential role of lexical diversity in written performance. For example,Crossley et al. (2011)found that lexical diversity predicted argumentative essay scores written by secondary students (Grades 9 and 11) and first-year university students, suggesting that it may be a useful measure when comparing students from different grade levels. However,McNamara et al. (2010)analyzed argumentative essays written by a less diverse sample (i.e., first-year university students only) and still found a positive relationship between lexical diversity and scores. Therefore, an alternate explanation is that genre may play a role. Whereas the two previous studies analyzed argumentative essays in which students draw on their personal knowledge and experiences to address the prompt (i.e., independent writing tasks), these business students were required to draw on specific readings, theories, and a case study in their responses (i.e., integrated or source-based writing). Prior research with English as a second language writers has shown that lexical diversity is higher in source-based writing because students incorporate lexical items from the sources into their essays (Gebril & Plakans, 2016). Since all students were required to apply the same key constructs and terms to the analysis of the same case, the range of vocabulary needed to accomplish the task may have been too restricted for lexical diversity to emerge as a predictor of their performance. Put simply, because the case study critique was text-specific, students had access to a similar vocabulary set when addressing the prompt.

The second language feature that failed to predict instructor scores was phrasal complexity, which is a key characteristic of academic writing (Biber et al., 2011) and has been associated with writing development (Crossley et al., 2011;Staples et al., 2016). Due to variation in the measures of phrasal complexity used in prior studies, it is possible that this methodological difference accounts for the divergent findings. For example,Biber et al. (2011)assessed phrasal complexity in terms of a set of grammatical features which included some features not included in the current study, whileCrossley et al. (2011)measured modifiers per noun phrase only. Our measure of phrasal complexity, complex nominals per clause as assessed by the Syntactic Complexity Analyzer (Lu, 2010), was previously used in a comparative study of argumentative essays written by university students (Lu & Ai, 2015) from different L1 backgrounds. They reported that English speakers produced a mean of 1.22 complex nominals per clause, which is similar to the mean reported here (1.19). It is possible that phrasal complexity may be less relevant for assessing variation within a more homogenous sample. In a study of L2 writing,Yoon (2017)reported that complex nominals per clause in argumentative essays differentiated among proficiency levels, but only for nonadjacent levels. In other words, it only captured variation in performance when more distant levels were compared. Genre differences between argumentative essays and disciplinary writing may help account for the divergent findings. It is possible that phrasal complexity is consistent when students produce essays with information from sources under time pressure and with length requirements, which is a context that requires disciplinary content to be conveyed concisely.

Implications

In the disciplinary business course examined here, students are exposed to core business theories and frameworks and are expected to apply that knowledge when critiquing case studies, which aligns with the association between case studies and business course assignments found in previous studies (Gardner & Nesi, 2013;Staples et al., 2018). Although the evaluation criteria specified content (75%) and essay structure (25%), the finding that error rate and lexical sophistication accounted for over 20% of the variance in scores suggests that instructors are sensitive to language when assessing student writing, which raises several potential pedagogical implications. It seems important to inform students that language is a factor in the evaluation of their written performance. To do so, however, instructors need to first identify their language expectations. For example, the evaluation criteria in this business course included the phrase “persuasive and vivid writing with proper structure of intro, paragraphs and conclusion,” which was worth only one of the five points allocated to essay structure. Considering that instructors had only one point to assess both “proper structure” and “persuasive and vivid writing,” it seems clear that language was not a priority in the stated grading criteria. Given the absence of any assessment criteria related specifically to language, it may be difficult for instructors to articulate what language features they consider when assessing the content of students’ answers. Due to the prevalence of research that has documented the types of language issues that are viewed negatively in the workplace (such as sentence fragments, run-on sentences, and grammatical errors), these features may be incorporated into assessment criteria.

For lexical sophistication, operationalized here as mean word length, instructors may find it useful to encourage students to use discipline-specific lexical items as opposed to generic words. Because disciplinary writing is about learning how to communicate about a defined body of knowledge concisely, the use of field-specific terminology is expected. In test-taking situations, instructors may suggest that students incorporate key phrases and terms from source texts. Students might benefit from instructor feedback to help them incorporate field-specific jargon into their discussions. Instructors could introduce students to a general academic word list (i.e., the Academic Word List,Coxhead, 2000), which contains a list of 570 academic word families that are widely used in texts across academic disciplines. Alternatively, they could provide a link toNelson’s (2000)academic word list for Business English, which has a list of words that frequently occur in business texts (https://users.utu.fi/micnel/business_english_lexis_site.htm).

Although these students wrote relatively accurately (94% to 95% error-free) while under time pressure, there are some potential pedagogical implications for the treatment of language errors in disciplinary classes. Even though faculty may perceive their role as teaching content and technical skills rather than teaching writing or language use (Neumann et al., 2019,Zhu, 2004a), it would not be overly burdensome to discuss frequent errors in a class period after exams or assignments have been graded and returned, such as by showing anonymous examples of errors and eliciting corrections. It might also be helpful to emphasize the importance of time management when carrying out writing tasks (both in academic courses and professional settings) so that students remember to save time for proof-reading before submitting their work. Including language resources on a course outline or in an online course management system can provide students with instructor-approved resources if they require additional information about specific structures or error types without requiring instructors to “teach” language.

Limitations and Future Research

Although the findings point to a clear relationship between language and instructor evaluation of business student writing even when assessment criteria emphasize content, it is important to note the limitations that impact the study’s generalizability. First, our study focused on a disciplinary business course at one university. While many features of the instructional setting seem typical for business courses described in the literature, replication studies in other contexts are necessary to confirm the role of language in exam performance. As described in the introduction, writing assignments in business courses include both professional writing as well as disciplinary content tasks. In the course examined here, the focus was on case study critiques as a tool for eliciting and evaluating disciplinary content. Additional research is needed to determine if language also plays a role in the content-oriented evaluation of other genres used to assess disciplinary knowledge (such as business proposals or design projects). Furthermore, studies with professional writing tasks (such as memos or letters) are needed to determine the role of language in successful workplace writing. To identify patterns in the evaluation of disciplinary writing, future studies should examine whether instructors are similarly influenced by language when assessing writing in other fields.

Our choice of language features was motivated by previous research that has documented the importance of errors, phrasal complexity, lexical diversity, and lexical sophistication in written texts that receive positive evaluations or were produced by more advanced writers. However, it is possible that additional language features also contribute to instructor evaluation of business writing. A more comprehensive approach that uses a wider range language features associated with academic writing would help instructors and researchers identify additional aspects of language that account for variation in the evaluation of student writing. Such studies should continue to explore language features identified in business writing specifically, as well as studies about academic writing generally. In light of our null findings for phrasal complexity and lexical diversity, future research should examine whether these language features play a role in instructor evaluation of different professional writing genres or disciplinary writing tasks. These language features may have a greater impact on the evaluation of student texts written without access to sources because students must supply their own sentence structures and vocabulary items, as opposed to sample from source-text language. Finally, the data in our study were fewer than 100 case study critiques (approximately 35,000 words), which is much smaller than a typical corpus linguistics study (500,000 words or more). Subsequent studies should aim to collect more samples of student writing to test the generalizability of the findings reported here.

Although we had some insight into instructors assessment practices through interviews from a previous study (Neumann et al., 2019), more research is needed to explore how instructors orient to language when assessing student writing for content. Despite the overt emphasis on disciplinary content, instructor scores in this context had clear links to language. Future studies might explore the potential discrepancies among evaluation criteria, instructor priorities when assessing student writing, and actual performance assessment. Especially in contexts where a core business course is taught by many instructors each year, it is important to ensure that evaluation criteria are uniformly interpreted and implemented. Research that includes think-aloud protocols during scoring or stimulated recall interviews after scoring is needed to further uncover how instructors define good disciplinary writing. Finally, data are also needed from students to determine whether their definitions of good disciplinary writing align with those of their instructors.

Conclusion

In the disciplinary business writing context investigated here, the evaluation criteria used to assess the students’ case study exams included content criteria related to business and leadership theory and essay structure, such as the presence of an introduction and conclusion and the use of claims and rebuttals of counter claims. Our analysis revealed, however, that 22% of the variance in instructor scores was predicted by language features that were not specified in the evaluation criteria. To ensure that students are aware of the covert role that language may play in the evaluation of disciplinary writing and that all students are evaluated equally, it is important to make all assessment criteria explicit. By explaining the language-related criteria in the task instructions and including them in the assessment criteria, business faculty can help students recognize the importance of language in disciplinary writing and develop writing skills needed in their immediate academic context and future workplace settings.

Footnotes

Acknowledgements

We would like to thank Nina Padden and the research assistants who helped with data collection, handling, and coding: Helene Bramwell, Bonnie Crawford, Diana Chojczak, Caroline Dault, Alexandre Kakon, Libing Lu, Dana Martin, Nora Sargent, and William Zullo.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this study was provided by grants awarded to the first author by the Canada Research Chairs program (Grant number 950-231218) and by funds from Dr. George Kanaan, Associate Dean for Student and Academic Affairs, John Molson School of Business, Concordia University.

Author’s Note

This study was approved by the Research Ethics Unit of Concordia University (Certificate No. 30005483).

ORCID iD

Kim McDonough

Author Biographies

Kim McDonoughis a professor of applied linguistics in the Education Department at Concordia University. Her current research projects are investigating visual cues in task-based interaction, collaboration in L2 writing, and written language development.

Pakize Uludagis a PhD candidate in applied linguistics in the Education Department at Concordia University. Her research interests include L2 writing, language assessment, and corpus linguistics.

Heike Neumannis a senior lecturer of English as a second language and a language test developer in the Education Department at Concordia University. Her research interests include second-language writing pedagogy, English for academic purposes, and language assessment.

References

AACSB International. (2020,July 28).2020 Guiding principles and standards for business accreditation.https://www.aacsb.edu/accreditation/standards/business

Beason

(2001).Ethos and error: How business people react to errors.College Composition and Communication,53(1),33-64.https://doi.org/10.2307/359061

Biber

(1988).Variation across speech and writing.Cambridge University Press.

Biber

Conrad

Reppen

Byrd

Helt

(2002).Speaking and writing in the university: A multidimensional comparison.TESOL Quarterly,36(1),9-48.https://doi.org/10.2307/3588359

Biber

Gray

Poonpon

(2011).Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly,45(1),5-35.https://doi.org/10.5054/tq.2011.244483

Biber

Johansson

Leech

Conrad

Finegan

(1999).Longman grammar of spoken and written English.Pearson Education Ltd.

Blanthorne

Bhamornsiri

Guinn

R. E.

(2005).Are technical skills still important? CPA Journal,75(3),64-65.

Boettger

R. K.

Emory Moore

(2018).Analyzing error perception and recognition among professional communication practitioners and academics.Business and Professional Communication Quarterly,81(4),462-484.https://doi.org/10.1177/2329490618803740

Brandenburg

L. C.

(2015).Testing the recognition and perception of errors in context.Business and Professional Communication Quarterly,78(1),74-93.https://doi.org/10.1177/2329490614563570

10.

Chan

C. S. C.

(2019).Long-term workplace communication needs of business professionals: Stories from Hong Kong senior executives and their implications for ESP and higher education.English for Specific Purpose,56(October),68-83.https://doi.org/10.1016/j.esp.2019.07.003

11.

Conrad

Newberry

(2011).24 Business communication skills: Attitudes of human resource managers versus business educators.American Communication Journal,13(1),4-23.http://ac-journal.org/journal/pubs/2011/spring/ConradNewberry.pdf

12.

Coxhead

(2000).A new academic word list.TESOL Quarterly,34(2),213-238.https://doi.org/10.2307/3587951

13.

Crossley

S. A.

Weston

J. L.

McLain Sullivan

S. T.

McNamara

D. S.

(2011).The development of writing proficiency as a function of grade level: A linguistic analysis.Written Communication,28(3),282-311.https://doi.org/10.1177/0741088311410188

14.

Durán

Malvern

Richards

Chipere

(2004).Developmental trends in lexical diversity.Applied Linguistics,25(2),220-242.https://doi.org/10.1093/applin/25.2.220

15.

Elrod

Tiggeman

Craven

A. E.

(2014).TSBPA’s business communications requirement: Has it made a difference? Journal of Case Studies in Accreditation and Assessment,3,1-13.https://www.aabri.com/manuscripts/131742.pdf

16.

Fraser

Harich

Norby

Brzovic

Rizkallah

Loewy

(2005).Diagnostic and value-added assessment of business writing.Business and Professional Communication Quarterly,68(3),290-305.https://doi.org/10.1177/1080569905279405

17.

Gardner

Nesi

(2013).A classification of genre families in university student writing.Applied Linguistics,34(1),25-52.https://doi.org/10.1093/applin/ams024

18.

Gardner

Nesi

Biber

(2019).Discipline, level, genre: Integrating situational perspectives in a new MD analysis of university student writing.Applied Linguistics,40(4),646-674.https://doi.org/10.1093/applin/amy005

19.

Gebril

Plakans

(2016).Source-based tasks in academic writing assessment: Lexical diversity, textual borrowing and proficiency.Journal of English for Academic Purposes,24(December),78-88.https://doi.org/10.1016/j.jeap.2016.10.001

20.

Ghannadian

F. F.

(2013).What employers want, what we teach.BizEd,12(2),40-44.https://bized.aacsb.edu/articles/2013/03/what-employers-want-what-we-teach

21.

Gimenez

(2017).Discipline-specific writing for business students: Research, practice and pedagogy. In Flowerdew

Costley

(Eds.),Discipline-specific writing: Theory into practice(pp.126-143).Routledge.

22.

Haswell

R. H.

(2000).Documenting improvement in college writing: A longitudinal approach.Written Communication,17(3),307-352.https://doi.org/10.1177/0741088300017003001

23.

Hutchins

T. D.

(2015).A new approach to teaching business writing: Writing across the core: A document based curriculum.American Journal of Business Education,8(2),131-138.https://doi.org/10.19030/ajbe.v8i2.9135

24.

Jackson

(2004).Case-based teaching in a bilingual context: Perceptions of business faculty in Hong Kong.English for Specific Purposes,23(3),213-232.https://doi.org/10.1016/j.esp.2003.05.001

25.

Larson-Hall

(2010).A guide to doing statistics in second language research using SPSS.Routledge.

26.

Leki

Carson

(1997).“Completely different worlds”: EAP and the writing experiences of ESL students in university courses.TESOL Quarterly,31(1),39-69.https://doi.org/10.2307/3587974

27.

(2010).Automatic analysis of syntactic complexity in second language writing.International Journal of Corpus Linguistics,15(4),474-496.https://doi.org/10.1075/ijcl.15.4.02lu

28.

(2015).Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds.Journal of Second Language Writing,29(September),16-27.https://doi.org/10.1016/j.jslw.2015.06.003

29.

Malvern

Richards

Chipere

Durán

(2004).Lexical diversity and language development.Palgrave Macmillan.

30.

Martin-Lacroux

Lacroux

(2017).Do employers forgive applicants’ bad spelling in résumés? Business and Professional Communication Quarterly,80(3),321-335.https://doi.org/10.1177/2329490616671310

31.

McNamara

D. S.

Crossley

S. A.

McCarthy

P. M.

(2010).Linguistic features of writing quality.Written Communication,27(1),57-86.https://doi.org/10.1177/0741088309351547

32.

Moshiri

Cardon

(2014).The state of business communication classes: A national survey.Business and Professional Communication Quarterly,77(3),312-329.https://doi.org/10.1177/2329490614538489.

33.

Nathan

(2013).Academic writing in the business school: The genre of the business case report.Journal of English for Academic Purposes,12(1),57-68.https://doi.org/10.1016/j.jeap.2012.11.003

34.

National Association of Colleges and Employers. (2019).The four career competencies employers value most.https://www.naceweb.org/career-readiness/competencies/the-four-career-competencies-employers-value-most/

35.

Nelson

(2000).Corpus-based study of business English and business English teaching materials. [Unpublished doctoral dissertation].University of Manchester.

36.

Nesi

Gardner

(2006).Variation in disciplinary culture: University tutors’ views on assessed writing tasks. In Kiely

Clibbon

Rea-Dickins

Woodfield

(Eds.),Language, culture and identity in applied linguistics(pp.99-117).Equinox.

37.

Neumann

Padden

McDonough

(2019).Beyond English language proficiency scores: Understanding the academic performance of international undergraduate students during the first year of study.Higher Education Research & Development,38(2),324-338.https://doi.org/10.1080/07294360.2018.1522621

38.

Nicolas

M. O.

Annous

(2013).Assessing WAC elements in business syllabi.Business and Professional Communication Quarterly,76(2),172-187.https://doi.org/10.1177/1080569912471709

39.

Plonsky

Oswald

F. L.

(2014).How big is “big”? Interpreting effect sizes in L2 research.Language Learning,64(4),878-912.https://doi.org/10.1111/lang.12079

40.

Polio

C. G.

(1997).Measures of linguistic accuracy in second language writing research.Language Learning,47(1),101-143.https://doi.org/10.1111/0023-8333.31997003

41.

Polio

C. G.

Shea

M. C.

(2014).An investigation into current measures of linguistic accuracy in second language writing research.Journal of Second Language Writing,26(December),10-27.https://doi.org/10.1016/j.jslw.2014.09.003

42.

Rogers

E. M.

(1994).A history of communication study: A biographical approach.Free Press.

43.

Schartel Dunn

S. G.

Lane

P. L

. (2019).Do interns know what they think they know? Assessing business communication skills in interns and recent graduates.Business and Professional Communication Quarterly,82(2),202-213.https://doi.org/10.1177/2329490619826258

44.

Sigmar

L. S.

Hynes

G. E.

(2012).Major difference: An examination of student writing performance by major and its implications for business communication.Administrative Issues Journal,2(1),133-146.https://dc.swosu.edu/aij/vol2/iss1/13

45.

Smart

K. L.

Hicks

Melton

(2013).Using problem-based scenarios to teach writing.Business and Professional Communication Quarterly,76(1),72-81.https://doi.org/10.1177/1080569912466256

46.

Staples

Biber

Reppen

(2018).Using corpus-based register analysis to explore the authenticity of high-stakes language exams: A register comparison of TOEFL iBT and disciplinary writing tasks.Modern Language Journal,102(2),310-332.https://doi.org/10.1111/modl.12465

47.

Staples

Egbert

Biber

Gray

(2016).Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre.Written Communication,33(2),149-183.https://doi.org/10.1177/0741088316631527

48.

Wardrope

W. J.

(2002).Department chairs’ perceptions of the importance of business communication skills.Business and Professional Communication Quarterly,65(4),60-72.https://doi.org/10.1177/108056990206500406.

49.

Weigle

S. C.

Friginal

(2015).Linguistic dimensions of impromptu test essays compared with successful student disciplinary writing: Effects of language background, topic, and L2 proficiency.Journal of English for Academic Purposes,18(June),25-39.https://doi.org/10.1016/j.jeap.2015.03.006

50.

Wolfe

Shanmugaraj

Sipe

(2016).Grammatical versus pragmatic error: Employer perceptions of nonnative and native English speakers.Business and Professional Communication Quarterly,79(4),397-415.https://doi.org/10.1177/2329490616671133

51.

Y. D.

Kendall

L. S.

(2006).Teaching faculty’s perspectives on business information literacy.Reference Services Review,34(1),86-96.https://doi.org/10.1108/00907320610648789

52.

Yoon

H.-J.

(2017).Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality.System,66(June),130-141.https://doi.org/10.1016/j.system.2017.03.007

53.

Zhu

(2004a).Faculty views on the importance of writing, the nature of academic writing, and teaching and responding to writing in the disciplines.Journal of Second Language Writing,13(1),29-48.https://doi.org/10.1016/j.jslw.2004.04.004

54.

Zhu

(2004b).Writing in business courses: An analysis of assignment types, their characteristics, and required skills.English for Specific Purposes,23(2),111-135.https://doi.org/10.1016/S0889-4906(02)00046-7