Influence of Group Composition on Participant Reactions to Training: A Study in an Indian Power Transmission Organization

Abstract

Investments in organizational training and its evaluation is important in recent times. One of the most popular models of training evaluation is the four-level model developed by Kirkpatrick. It includes participant reactions (Level I), extent of learning (Level II), the extent of transfer of training through appropriate behaviour (Level III) and improvement in organization performance (Level IV). Despite its overwhelming use in the industry, organizations have frequently gathered data on Level I of this model only. While extant literature researched on factors affecting participant reactions; group composition influences merited scant attention. To address this gap, the current study conducted at an Indian power transmission organization, focused on the influence of group composition on participant reactions to training of a programme titled ‘Empowering Self for Better Performance’. Reaction data were collected from all the 120 participants who attended the programme. Data analyses pointed out to significant resultant differences in perception on two major dimensions of reaction—programme content and programme duration among participants with differences in age, organizational tenure, job position and educational qualifications. Our results not only provide empirical credence to the importance of group composition in influencing participant reactions but also bear implications for training design of millennials and soft skills programmes in heterogenous groups.

Keywords

Training training evaluation Kirkpatrick model age participant reactions

Introduction

Organization training has been considered as one of the most widespread methods to enhance individual as well as organizational productivity (Tan, Hall, & Boyce, 2003). Given the resources invested in training coupled with the fact that training today is either outsourced or made mandatory in most organizations, its systematic monitoring and evaluation has gained tremendous importance (Giangreco, Sebastiano, & Peccei, 2009). Hence, though evaluation of training programmes may pose a difficulty (Lechner, 2000), organizations recognize its necessity and their major issue is identifying appropriate training evaluation models.

With due acknowledgement of several approaches/models to evaluate training programmes (Kraiger, Ford, & Salas, 1993), most training evaluations continue to use the popular and classic model developed by Kirkpatrick (1959a, 1959b, 1960a, 1960b). The efficacy of training endeavours has traditionally been measured using outcomes deemed relevant by organizations (Galloway, 2005). However, these outcomes need measurement on several criteria to avoid any biased conclusions. To minimize such bias, Kirkpatrick (1996) identified four levels for evaluating training programmes. Each of these levels was hierarchical and theorized to influence the next. They include the immediate participant reactions to the training itself (Level I), the extent of learning from the programme (Level II), the extent to which the new training transfers back onto the job and results in new forms of behaviour (Level III) and finally the extent to which these new behaviours result in improved individual and organizational performance (Level IV) (Giangreco et al., 2009). Kirkpatrick’s hierarchical model of training evaluation was a seminal contribution and resulted in development of newer evaluation models, publications oriented to practitioners, debate and robust criticisms on the subject. Despite such notable contributions of this model, its real application in the industry was primarily limited to the first level, that is, reaction (Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Morgan & Casper, 2000; Plant & Ryan, 1994; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008). As data on Levels II–IV of this model was expensive and difficult to collect, training participants’ reactions has been the commonly used criterion for evaluating the efficacy of training programmes (Lee & Pershing, 2002).

While Kirkpatrick and more recent researchers (Arthur, Bennett, Edens, & Bell, 2003) maintain that participant reactions are not surrogate measures of training efficacy, a careful examination of such reactions and the factors influencing them can help in better design and management of training programmes (Giangreco et al., 2009). Given the importance accorded to participant reactions in training by practitioners and organizations, extant research is yet to fully explore the breadth of factors influencing this primary level of training evaluation (Hook & Bunce, 2001; Sitzmann et al., 2008). The purpose of the current study is to understand the influence of group composition on participant reactions to training of a programme, in an Indian power transmission organization. The importance of organizational training and its subsequent evaluation has gained momentum in Indian organizations in the past 15 years (Sharma, 2014). Indian organizations have slowly and steadily recognized training as a strategic rather than tactical priority (Jain & Agrawal, 2005). With specific reference to our research context, India has a vast network of transmission and distribution of power and yet experiences one of the highest losses in both these domains. Organizations in this sector have responded not only by upgrading their transmission and distributions systems but also by harnessing their human resources through training (Ghosh, Joshi, Satyawadi, Mukherjee, & Ranjan, 2011). In the backdrop of a need for training in the chosen research context, our aim in specific is to understand the influence of age, education, job position and organizational tenure of participants on their reactions to training.

Review of Literature

Importance of Reaction Evaluation in Training

Reaction evaluation is an immediate assessment following training and is usually collected through questionnaires after a training programme, conclusion of a major segment of the programme or after a section that has undergone considerable revision during its design (Lee & Pershing, 2002; Pont, 1996). It provides an opportunity to participants to evaluate the relevance of the topic of training, the quality of materials used, the quality of the instructor, the duration of the course and logistics arrangement among other aspects (Galloway, 2005). Morgan and Casper (2000) have provided three broad uses of participant reactions. First, credible reaction data can immeasurably help in diagnosing the quality of design and delivery of training. Second, the collection of participant reactions may emphasize the seriousness of the training function. Finally, participant reactions might act as potential predictors of the other levels of Kirkpatrick’s model of evaluation. Though Alliger and Janak (1989) and Alliger et al. (1997) in their noteworthy and often cited meta-analysis on relations between the four levels of evaluation proposed by Kirkpatrick clearly pointed out the lack of systematic or nomological relationships between reaction and any other level, dimensions of reactions nevertheless retain importance. These dimensions referred to as utility judgements (Alliger et al., 1997) reflect the extent to which participants perceive the application of the training content in their immediate jobs.¹ Utility judgements in participant reactions have enjoyed significant relationships to immediate learning and transfer (Morgan & Casper, 2000).

Extant literature on training evaluation has provided two conclusions important for this study. First, organizations face difficulty in evaluating their training programmes at all the four levels (Giangreco et al., 2009; Plant & Ryan, 1992), thereby pointing out an interesting dichotomy between academic research on Kirkpatrick’s model and its real application in practice. Second, as mentioned in the introductory session, participant reactions are the commonest measure of training evaluation. Organizations participating in the American Society for Training and Development’s benchmarking forum reported evaluating 91 per cent training programmes using reaction data (Sugrue & Rivera, 2005). Yadapadithaya and Stewart (2003) in a massive cross-national study of organizations in India and Britain made similar observations. Their study revealed that nearly all of the Indian and British organizations surveyed by them conducted the first level of training evaluation while nearly 74 per cent of Indian organizations conducted the second level of evaluation, that is, learning. Levels III and IV of evaluation were conducted by a negligible proportion of organizations. Studies worldwide reported distinctly low proportions of companies conducting all the prescribed four levels of evaluation (Arthur et al., 2003; Olsen, 1998; Van Buren & Erskine, 2002). Thus, the importance of participant reactions coupled with its possible use to organizations urges for a better understanding of this data and more importantly factors influencing this primary level of evaluation.

Factors Influencing Participant Reactions in Training Evaluation

Academic research has had a definite focus on the correlation between participant reactions and the other levels of evaluation and also provided contrasting conclusions (Alliger et al., 1997; Arthur et al., 2003; Clement, 1982; Russ-Eft & Preskill, 2001). But similar attention has unfortunately not been provided to the factors influencing participant reactions to training. Factors that influence reaction evaluations are important and merit an independent examination apart from whether participant reactions influence the other levels of evaluation or not (Giangreco et al., 2009).

Sitzmann et al. (2008) in a comprehensive meta-analysis of studies assessing participant reactions divided these factors into two broad categories, namely, trainee characteristics and situational characteristics. This meta-analysis revealed that four trainee characteristics have been examined regularly as antecedents to participant reactions, namely, (a) pre-training motivation, (b) mastery goal orientation, (c) trainee agreeableness and (d) anxiety. Pre-training motivation in trainees has shown significant correlations with training. Similarly, mastery goal orientation or tendency of a trainee to learn, become self competent and overcome obstacles to learning is also positively correlated with his/her overall learning and satisfaction with the training experience (Kozlowski et al., 2001; Payne, Youngcourt, & Beaubien, 2007). Trainees with high agreeableness tend to display a more positive outlook to training and often sympathize with minor inconveniences during training execution than others, whereas anxiety correlated negatively with trainee reactions during training (Sitzmann et al., 2008). Situational characteristics such as instructor style, human interaction during training, perceived trainer performance, perceived efficiency of training, perceived use of training and organizational support also significantly influenced participant reactions. Instructor style labelled as immediacy or the verbal and nonverbal behaviours to reduce the psychological distance between the trainer and participants positively influenced participant reactions (Allen, Witt, & Wheeless, 2006). Perceived organization support, perceived trainer performance (both in content and in pedagogy) and increased human interaction during training enhanced overall participant satisfaction with the training experience (N.J. Entwistle & A. Entwistle, 1991). In a more recent study, Diamantidis and Chatzoglou (2014) concluded that robust training design ensured trainee satisfaction as well as impacted on the post-training performance of trainees. Perceived efficiency of training correlated positively with participant reactions and included factors such as optimal arrangement of logistics for the programme, proper scheduling of training sessions and the quality of teaching materials used (Giangreco et al., 2009; Lee & Pershing, 2002). Perceived use of training reflected in the extent to which training equipped participants with skills necessary for the current job or future career prospects within the organization also helped increase overall satisfaction with the programme (Velada & Caetano, 2007). In the Indian context, a recent study by Ghosh, Satyawadi, Joshi, Ranjan and Singh (2012) concluded that a trainer’s comfort with the subject matter and his/her interpersonal skills with the trainees ensured trainee satisfaction.

An interesting cache of research on diversity training² focused on trainee demographics and group composition as relevant factors influencing the effectiveness of training inclusive of participant reactions (Bezrukova, Jehn, & Spell, 2012; Roberson, Kulik, & Pepper, 2001). Roberson et al. (2001) researching on the influence of group composition on the diversity training initiatives concluded that while group composition did not share a direct relationship with training outcomes, it did have an interactive effect with certain outcome measures such as that of prior diversity training experience. Group heterogeneity was beneficial for trainees possessing little prior diversity training experience. Bezrukova et al. (2012) in a critical examination of 178 articles on trends in diversity training pointed out that only 17 studies had focused on the effect of training group composition inclusive of trainee demographics (such as age, race and gender) on outcomes of diversity training. This examination also indicated that race as a demographic variable had received maximum attention from researchers herein while age of trainees was largely ignored in extant research (with few notable exceptions such as that of Bertolino, Truxillo, & Fraccaroli, 2011). Bertolino et al. (2011) concluded that age did play a moderating influence on the relationship between proactive personality with training motivation, training behavioural intentions and perceived career development from training.

Justification for the Current Research and the Study Context

Despite the impressive array of extant research on participant reactions in training evaluation, certain research gaps exist germane to the purpose of our study. First, extant research unanimously accorded importance to the relative ease in collecting participant reactions in comparison to the other levels in Kirkpatrick’s evaluation framework, its practical utility for organizations and hence urged for a better understanding of this phase, that is, Level I in training evaluation. Our background review of literature indicates a paucity of research on the spectrum of factors influencing participant reactions (Giangreco et al., 2009). Second, and more importantly, systematic meta-analysis of literature (as recorded in the previous sub-section) stated a focus on specific individual and situational factors influencing reactions. Surprisingly, group composition including participant demographics has rather been neglected in studies herein. Group composition (participant age, race, gender, etc.) merited limited attention in literature pertaining to outcomes of diversity training initiatives in organizations but not with specific relation to participant reactions in training evaluation of varied training programmes. It may be noted, however, that age, job position, job tenure and educational qualification of participants have been studied in correlation with learning (Level II) and transfer of training (Level III) of training evaluation (Lim & Morris, 2006; Tannenbaum, Mathieu, Salas, & Cannon-Bowers, 1991). Thus, in the backdrop of these predominant research gaps, we conclude that influence of group composition on participant reactions to training presents a much needed and potentially insightful avenue for research. The major all-encompassing objective of the current study is to investigate the influence of group composition on various parameters of participant reactions. Group composition in our study has included age, educational qualifications, organization tenure and job position of trainees/participants.

The current study has been carried out in one of the largest power transmission companies in India. The registered office is situated at Bhubaneswar, the capital of the State of Odisha, India. Its projects and field units are spread all over the State. Training, both in technical and non-technical/soft skills, was a regular feature in the company with the main training centre located at Bhubaneswar. One of the most popular training programmes conducted in the company was titled as ‘Empowering Self for Better Performance’. The major contents of the programme included (a) significance of empowering self, (b) power of positive attitude, (c) assertive communication, (d) self-discipline and self-audit and (e) change management. The programme spread over 2 days and facilitated by two instructors was conducted in eight batches with an average batch size of 15 participants. Given the generic nature and content of the programme, it could cut across participants in differing age groups, educational qualifications, job positions, field locations and organization tenure. Hence, the programme was considered ideal for the current study. Participant reaction data were collected immediately the conclusion of the programme.

Methodology

Sample

A total of 120 reaction forms were collected from participants on conclusion of all the eight batches of the programme. After initial screening of the reaction data, eight responses were rejected as they were incomplete. Further, few outliers were detected through box plots, that is, (Respondent Nos: 15, 22, 23, 24, 27, 58, 62, 76, 77, 80, 84, 91, 100, 109) and removed from the database. Such activities resulted in a final sample size of 98 accurate responses for our study. Table 1 presents a profile of the final list of respondents/participants of the programme. As it is evident from the table, the participants were predominantly males, between age group of 26–35 and with less than 5 years of experience. However, the proportion of participants’ education level was same across three levels with negligible numbers completing their post-graduation degree.

Measures

The questionnaire used to collect participant reaction data was developed by the organization in consul-tation with the researchers. The intent was to develop a standardized tool for use in subsequent batches of the same programme. Given the purpose of our study, the section on demographic information of the participants was deemed as important. Participants’ age, gender, job position, organizational tenure and education were collected using a multiple-choice response format. Several sources were used to develop the initial dimensions of reaction evaluation such as that of extant literature and participant reaction forms of similar such training programmes conducted by benchmarked organizations. The initial questionnaire was based on 11 dimensions of reaction evaluation as summarized by Lee and Pershing (2002), namely, programme content, programme materials, delivery methods, instructor/facilitator, instructional activities, programme duration, training environment, planned action/expectation, logistics, overall evaluation and recommendations for further improvement. Subsequent discussions between the organization and the researchers indicated that this initial questionnaire needed brevity and clarity for its proper completion by participants. The final seven dimensions of reaction evaluation in the questionnaire included (a) programme content, (b) delivery methods and multimedia, (c) instructor/facilitator efficacy, (d) programme duration, (e) logistics, (f) overall evaluation and (g) an open-ended question on recommendations for further improvement. Responses were generated on a six-point rating scale where ‘1’ denoted ‘poor’ and ‘6’ denoted ‘excellent’ for all dimensions save programme duration and recommendation for improvement. Programme content contained four items (α = 0.71) while delivery methods and visual aids included four items (α = 0.81). Finally, instructor/facilitator efficacy was measured through four items (α = 0.87). The overall reliability of the questionnaire was α = 0.75. Our discussions with the training department increased the specificity, appropriateness and face validity of the questionnaire. Reliance on extant literature to develop the dimensions of reaction evaluation ensured content validity of the final questionnaire.

Table 1.

Sample Profile

Group Composition/ Demographic Variables	Categories	Frequency (%)
Job position*	Executives	52
	Non-executives	48
Gender	Male	76.5
	Female	23.5
Age (years)	Below 25	9.2
	26–35	52.0
	36–45	10.2
	45 above	28.6
Education	1 (Intermediate, 10, +2)	33.7
	2 (ITI/Diploma**)	22.4
	3 (bachelors’ degree)	38.8
	4 (masters’ and above)	5.1
Experience/tenure (Years)	1 (less than 5 years)	63.3
	2 (6–10 years)	5.1
	3 (more than 10 years)	31.6

Source: Authors’ own findings.

Notes: * Job positions were divided into two broad categories as indicated in Table 1. Executives include engineers with a diploma/bachelor’s degree/master’s degree. They also included personnel in human resources, information technology, legal division, public relation, medical services, library sciences and administration with a minimum of a bachelor’s degree. Non-executives included personnel with an educational qualification of matriculation or an intermediate degree.

**The industrial training institutes (ITI) under the Directorate General of Employment and Training (DGET), Government of India, provide training and certification after an intermediate degree in various trades and specialities pertaining to engineering.

Data Collection Procedure

Potential participants to the programme were nominated by the main training centre, based on the results of their training need analysis and in consultation with the heads of their respective departments. A total of 160 nominations across seven different field locations (Cuttack, Paradeep, Jagatsinghpur, Kalahandi, Puri, Koraput and Rourkela) were received for the programme, and it commenced from January 2016 with due approval from the management. With an average completion of one batch per month and analysis of participant reaction responses by the training department, the subsequent batches were conducted within September of 2016 at the main training centre, Bhubaneswar. In the ultimate analysis, considering participant availability from their field destinations, 120 employees attended the programme. Participant response to the reaction questionnaire was anonymous as there was no identifying information such as that of name and employee number. As participants were assured of the confidentiality of their responses they returned the filled in questionnaires directly to the researchers after their programme.

Results

The ANOVA tests, t-tests and a two-stage cluster analysis were used to analyse the reaction responses gathered. The descriptive statistics presented in Table 2 indicated that though substantial number (54%) of participants provided good feedback on programme content, there existed a scope for further improvement in comparison to participant responses on dimensions of delivery methods and instructor efficacy. The overall grading of the programme was positive and bulk of the participants (61.2%) felt that the length/duration of the programme was inadequate or less.

To ascertain whether participants exhibited differences in their response to the chosen dimensions of reaction evaluation, a t-test (Table 3) was conducted. The resultant findings revealed that participants in different job positions, that is, executive and non-executive differed significantly on three reaction dimensions mainly programme content (t = 10.41; p < 0.001), programme duration (t = 5.75; p < 0.001) and logistics (t = 3.133; p < 0.01). Non-executive participants were satisfied with the programme content but rather dissatisfied with the programme duration and arrangement of logistics in comparison to executive participants. These results also helped us infer the overall satisfaction of participants across job positions with the programme, instructor/facilitator efficacy and delivery methods.

The results of the ANOVA tests are presented in Tables 4–6. These tests were carried out to find out differences if any in participant responses of different demographic groups on the dimensions of reaction evaluation. Out of all the dimensions of reaction evaluation as provided in Table 2, significant differences in mean scores were observed mainly for two, that is, programme content and programme duration. Results provided in Table 4 pointed out to differences in responses of participants belonging to different age groups on, namely, programme content (F = 8.526; df = 3/94; p < 0.001) and programme duration (F = 10.054; df = 3/94; p < 0.001). Participants belonging to age groups 36–45 and 46 and above expressed their satisfaction with the programme content in comparison to their counterparts in other age groups. In contrast, participants below 25 years of age significantly differed in their perceptions about programme duration and were satisfied with the length of the programme in comparison to participants above 35 years of age.

These results also were interrelated to the pattern of responses on dimensions among participants with differences in organizational tenure (Table 5). Participants with more than 10 years of experience/tenure in the company also belonged to the older age category, that is, more than 35 years. They differed significantly in their reaction scores on satisfaction with the programme content (F = 11.419; df = 2/95; p < 0.001) as compared to younger participants (i.e., 25 years of age) and with an organizational tenure of less than 5 years. The former also differed in their perception about the programme duration (F = 12.202; df = 2/95; p < 0.001) and logistics (F = 3.204; df = 2/95; p < 0.01) in comparison to the latter. While participants with lesser experience/tenure were happier with the programme duration in comparison to their more experienced counterparts, they expressed less satisfaction with the programme content. When reaction scores of participants with different educational qualifications were compared (Table 6), they differed mainly on two dimensions, that is, programme content (F = 10.729; df = 3/94; p < 0.001) and programme duration (F = 5.13; df = 3/94; p < 0.001). Mean scores (Table 6) revealed that participants possessing an intermediate degree were far more satisfied with the programme content and duration in comparison to others with an ITI/diploma or a masters’ degree.

Based on the results of the ANOVA tests as provided in Tables 4–6, recurring differences in mean scores were found primarily in two dimensions of reaction evaluation, that is, programme content and programme duration. A further classification of participants across age, educational qualification and organizational tenure with respect to their response on these two dimensions of reaction evaluation was conducted through a two-stage cluster analysis. Cluster analysis has been traditionally employed as a legitimate statistical method for classification as it makes no prior assumptions about important differences within a population (Punj & Stewart, 1983). Given the purely empirical and inductive stance of this method, its use herein yielded the presence of two discrete categories of participants (Table 7). For a better illustration of these categories or clusters identified, monikers have been provided. These results also simulated the pattern of responses presented earlier in Table 3. Cluster 1 comprised mainly of executives while cluster 2 represented the non-executive job position.

Cluster 1: Millennials

The first of the identified clusters contained 51 participants with less than 5 years of organizational tenure and primarily in the age group of 26–35 years. Majority of these participants also possessed a bachelors’ degree (52.9%). This group of young though better educated participants expressed less of satisfaction with the programme content but rated positively on the programme duration as indicated by the mean scores in Table 7. The cluster was named as millennials based on their youth, brevity of organizational experience and yet higher educational qualifications (Gursoy, Maier, & Chi, 2008).

Cluster 2: Traditionalists

47 participants in the second cluster were considerably older, that is, above 46 years, less educated (70.2% of participants possessed an intermediate degree) and had been employed with the organization for a longer duration, that is, 10 years or more. We named this cluster as traditionalists. This name akin to Generation X, signified individuals above 40 years of age, and with less of technology immersion than millennials (Sandeen, 2008). These participants were happy with the programme content but expressed a relative dissatisfaction with perceived brevity in programme duration.

Table 2.

Descriptive Statistics (Frequency %)

Reaction Evaluation Dimensions	Excellent	Very Good	Good	Satisfactory	Unsatisfactory	Poor
Programme Content
1. Coverage of programme content as specified	31.6	23.5	28.6	16.3	0	1.0
2. Relevance of content	33.7	22.4	27.6	16.3	0	0
3. Clarity of sessions	57.1	39.8	3.1	0	0	0
4. Utility of the programme content in the current job	56.1	40.8	3.1	0	0	0
Delivery Methods and Multimedia
1. Use of multimedia in delivery of content	52.0	37.8	10.2	0	0	0
2. Clarity and specificity of delivery methods	45.9	35.7	18.4	0	0	0
3. Impact of delivery methods	48	40.8	11.2	0	0	0
Instructor/Facilitator Efficacy
1. Domain knowledge of instructor/facilitator	61.2	35.7	3.1	0	0	0
2. Standard of delivery of content	52.0	41.8	6.1	0	0	0
3. Ability to demonstrate with relevant examples	43.9	46.9	43.9	0	0	0
4. Ability to stimulate and control discussion	43.9	46.9	9.2	0	0	0
Logistics
Quality of venue/administrative arrangement	38.8	44.9	16.3	5.7	3.8	1.0
Over all Grading of the Programme	41.8	55.1	3.1	0	0	0
Programme Duration	Less		Adequate	More
Length of the programme	61.2		37.8	1.0

Source: Authors’ own findings.

Table 3.

Comparative Responses of Participants in Different Job Positions on Reaction Evaluation Dimensions

Reaction Evaluation Dimensions	Job Position	N	Mean	t-Test
Programme Content	Executive	51	18.74	10.415***
Programme Content	22.43
Adequacy of programme duration	Executive	51	2.24	5.754***
Adequacy of programme duration	1.26
Logistics	Executive	51	5.43	3.133**
	Non-executive	47	5.00

Source: Authors’ own findings.

Notes: **p < 0.01; ***p < 0.001.

Table 4.

Differences in Responses across Age Groups on Reaction Evaluation Dimensions

Reaction Evaluation Dimensions	Age	N	Mean	SD	F
Programme content	<25		18.8889	1.83333	8.526***
	26–35	51	19.7255	2.56966
	36–45	10	21.9000	2.28279
	>46	28	21.9643	1.79469
	Total	98	20.5102	2.53733
Length of the programme	<25		2.78	0.667	10.054***
	26–35	51	1.98	1.010
	36–45	10	1.30	0.675
	>46	28	1.21	0.630
	Total	98	1.77	0.972

Source: Authors’ own findings.

Notes: F = one-way ANOVA, degrees of freedom (df) = 3/94, ***p < 0.001.

Table 5.

Differences in Responses across Experience/Organizational Tenure Groups on Reaction Evaluation Dimensions

Reaction Evaluation Dimensions	Experience/Tenure	N	Mean	SD	F
Programme content	1.00	62	19.7097	2.49844	11.419***
	2.00		20.4000	2.30217
	3.00	31	22.1290	1.83924
	Total	98	20.5102	2.53733
Length of the programme	1.00	62	2.08	0.997	12.202***
	2.00		1.80	1.095
	3.00	31	1.13	0.499
	Total	98	1.77	0.972
Logistics	1.00	62	5.35	0.630	3.204**
	2.00		4.80	0.837
	3.00	31	5.03	0.795
	Total	98	5.22	0.711

Source: Authors’ own findings.

Notes: F = one-way ANOVA, degrees of freedom (df) = 2/95, 1.00: <5 years, 2.00: 6–10 years, 3.00: >10 years, **p < 0.01, ***p < 0.001.

Table 6.

Differences in Responses across Educational Qualification Groups on Reaction Evaluation Dimensions

Reaction Evaluation Dimensions	Education	N	Mean	SD	F
Programme content	1.00	33	22.18	1.87	10.729***
	2.00	22	19.40	2.40
	3.00	38	20.02	2.44
	4.00		18.00	1.58
	Total	98	20.51	2.53
Length of the programme	1.00	33	1.30	0.728	5.135***
Length of the programme	2.05	0.999
	3.00	38	1.89	1.00
	4.00		2.60	0.89
	Total	98	1.77	0.97

Source: Authors’ own findings.

Notes: F = one-way ANOVA, degrees of freedom (df) = 2/95, 1.00: Intermediate (10 + 2), 2.00: ITI/diploma, 3. 00: bachelors’ degree, 4.00: masters’ and above, ***p < 0.001.

Table 7.

Two-stage Cluster Analysis of Participants Based on Responses to Programme Content and Duration

Cluster	1	2
Description	Millennials	Traditionalists
Size	52.0% (51)	48.0% (47)
Inputs	Experience/tenure: <5years (100%)	Experience/tenure: >10 years (66%)
	Education level graduate: (52.9%)	Intermediate: (70.2 %)
	Age: 26–35 years (74.5%)	Age: >46 years (59.6%)
Evaluation field	Programme content Mean: 19.06	Programme content Mean: 22.09
	Programme Duration Mean: 2.16	Programme duration Mean: 1.34

Source: Authors’ own findings.

Discussions and Implications

The major purpose of this study, that is, to examine the influence of group composition (inclusive of age, job position, educational level and organizational tenure) on participant reactions in training, was derived from two major trends noticeable in extant literature. The first of these included the divide between academia and practitioners in the actual use of Kirkpatrick’s model. While practitioners focused primarily on Level I as mentioned earlier in the article; academicians emphasized upon moving the analysis of training evaluations beyond Level I (Giangreco et al., 2009). This dichotomy as noted in our review of extant literature, diluted a systematic understanding of evaluation data readily available in organizations, that is, participant reactions and the factors influencing participants’ overall satisfaction with training. The second trend that reinforced the purpose of this study stemmed from the lack of studies focusing on the spectrum of factors influencing participant reactions. While the influence of several individual and situation characteristics was studied, group composition influences on participant reaction had been ignored. Our study sought to address these trends by specifically examining the influence of group composition variables (Table 1) on differing dimensions of reaction evaluation. Within the context of the ‘empowering self’ programme, our findings did point out majorly to certain recurring differences in perceptions on two dimensions of reaction evaluation: programme content and programme duration. While group composition of participants did not significantly influence all the chosen dimensions of reaction evaluation in our study, our findings nevertheless provided credence to the importance of particularly age, organizational tenure and educational level in generating a pattern to reaction scores on the aforementioned dimensions. These findings also enrich the limited literature (mainly in the context of diversity training, e.g., Bezrukova et al., 2012) existing with a focus on trainee demographics/group composition and their influence on participant reactions.

The results of the cluster analysis (a culmination of results submitted in Tables 3–6) contribute to extant work as age of participants provided one of the more important basis for classification of clusters (Table 7). Age of participants was also associated with their organizational tenure and educational qualifications. Earlier research, as recounted in the article, had very limited focus on age regarding participant reactions. But our findings highlight the importance of age in inducing a pattern to participant reactions on the said dimensions of participant reactions. They also provide interesting implications for training programme design and content for millennials. Items measuring reaction scores on programme content (Table 2) particularly those pertaining to coverage of content as specified and relevance of programme content indicated a scope for improvement. The facilitators of the programme had collated necessary content into an instruction manual and distributed it to participants prior to the commencement of training. Despite these initiatives, millennials, as named in our study, expressed dissatisfaction with the programme content in comparison to their older peers. These findings concur with earlier literature focusing on training for the current generation (Oblinger, 2003). In the backdrop of technology infusion, millennials are described as nonlinear in their learning style relying less on course materials, instruction manuals and documents (Feiertag & Berge, 2008). They enjoy the hands-on process of trial and error learning and seek relevance from the learning context into their work and day-to-day life (Shaw & Fairhurst, 2008). With greater cognizance of this ‘hypertext’ mindset (as aptly termed by Feiertag & Berge, 2008), training initiatives call for use of engaging and interactive multimedia in design of training content rather than detailed instruction manuals (Shaw & Fairhurst, 2008). But over and above the use of technology, millennials need an absolute conviction of the utility or relevance of training in the real world (Partridge & Hallam, 2006).

Millennials and traditionalists also differed in their reaction scores on programme duration. The ‘empowering self’ programme was conducted over 2 days. Millennials were satisfied with this time span, whereas traditionalists perceived it as brief. While not providing any direct linkages with extant literature on training, this difference nevertheless does help us explore possibilities with research on age and time perception psychology (Ferreira, Paiva, Prando, Graça, & Kouyoumdjian, 2016). Ferreira et al. (2016) in an experimental study lucidly depicted that time elapsed faster for older individuals in the age group of 40 and above in comparison to younger participants. The perception of time was moderated by new experiences, awe, fear and age (Rudd, Vohs, & Aaker, 2012). These studies may not bear any immediate implications herein but open an exciting window of potential for research on deciding the optimal length/duration of similar training programmes for differing age groups. As rapidity of time increased with age, longer duration training might be appreciated by traditionalists than millennials.

Linkages of these results with literature in varied domains also provide implications for design training on soft skills. The ‘empowering self’ programme was categorized as non-technical and behavioural training by the concerned organization. The first of these implications relate to the difficulty in building relevance to soft skills training as the degree of similarity between training, work and work environment is relatively less in comparison to hard skills or technical training (Laker & Powell, 2011). Hence, trainers need to grasp the need for such skills in a specific organizational context as these skills are often employer defined and locally relevant (Grugulis & Vincent, 2009). To build better relevance to programmes as the ‘empowering self’, trainers need to specify the outcomes and their potential in the organization itself prior to the training. Second as training groups depend on participant availability and requirements they are often heterogenous in their composition. As this study succinctly points out to the impact of group composition on participant reactions; trainers need to cater to unique requirements of differing groups of participants. This need is even pronounced in the case of soft skills training in the absence of immediacy and salience of training on the job (Laker & Powell, 2011).

Despite the implications of this study, there exist certain limitations as well. The sample of the study is relatively small due to unavoidable reasons as mentioned beforehand in the article. Moreover, our study is limited to an organizational context. The inherent strength of the study is in researching an area (i.e., group composition) which had hitherto been neglected in the ongoing study on participant reactions. The insights gained from the study can be used in the designing similar soft skill programmes for heterogenous training groups in organizations in differing industries.

Footnotes

Notes

References

Allen

, Witt

P. L.

, & Wheeless

L. R.

(2006). The role of teacher immediacy as a motivational factor in student learning: Using meta-analysis to test a causal model. Communication Education, 55(1), 21–31.

Alliger

G. M.

, & Janak

E. A.

(1989). Kirkpatrick’s levels of training criteria: Thirty years later. Personnel Psychology, 42(2), 331–342.

Alliger

G. M.

, Tannenbaum

S. I.

, Bennett

Jr. , Traver

, & Shotland

(1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341–358.

Arthur

Jr. , Bennett

Jr. , Edens

P. S.

, & Bell

S. T.

(2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234–245.

Bertolino

, Truxillo

D. M.

, & Fraccaroli

(2011). Age as moderator of the relationship of proactive personality with training motivation, perceived career development from training, and training behavioral intentions. Journal of Organizational Behavior, 32(2), 248–263.

Bezrukova

, Jehn

K. A.

, & Spell

C. S.

(2012). Reviewing diversity training: Where we have been and where we should go. Academy of Management Learning & Education, 11(2), 207–227.

Clement

R. W.

(1982). Testing the hierarchy theory of training evaluation: An expanded role for trainee reactions. Public Personnel Management, 11(2), 176–184.

Diamantidis

A. D.

, & Chatzoglou

P. D.

(2014). Employee post-training behaviour and performance: Evaluating the results of the training process. International Journal of Training and Development, 18(3), 149–170.

Entwistle

N. J.

, & Entwistle

(1991). Contrasting forms of understanding for degree examinations: The student experience and its implications. Higher Education, 22(3), 205–227.

10.

Feiertag

, & Berge

Z. L.

(2008). Training generation N: How educators should approach the net generation. Education + Training, 50(6), 457–464.

11.

Ferreira

V. F. M.

, Paiva

G. P.

, Prando

, Graça

C. R.

, & Kouyoumdjian

J. A.

(2016). Time perception and age. Arquivos de neuro-psiquiatria, 74(4), 299–302.

12.

Galloway

D. L.

(2005). Evaluating distance delivery and e-learning is kirkpatrick’s model relevant? Performance Improvement, 44(4), 21–27.

13.

Ghosh

, Joshi

J. P.

, Satyawadi

, Mukherjee

, & Ranjan

(2011). Evaluating effectiveness of a training programme with trainee reaction. Industrial and Commercial Training, 43(4), 247–255.

14.

Ghosh

, Satyawadi

, Joshi

J. P.

, Ranjan

, & Singh

(2012). Towards more effective training programmes: A study of trainer attributes. Industrial and Commercial Training, 44(4), 194–202.

15.

Giangreco

, Sebastiano

, & Peccei

(2009). Trainees’ reactions to training: An analysis of the factors affecting overall satisfaction with training. The International Journal of Human Resource Management, 20(1), 96–111.

16.

Grugulis

, & Vincent

(2009). Whose skill is it anyway? ‘soft’ skills and polarization. Work, Employment and Society, 23(4), 597–615.

17.

Gursoy

, Maier

T. A.

, & Chi

C. G.

(2008). Generational differences: An examination of work values and generational gaps in the hospitality workforce. International Journal of Hospitality Management, 27(3), 448–458.

18.

Hook

, & Bunce

(2001). Immediate learning in organisational computer training as a function of training intervention affective reaction, and session impact measures. Applied Psychology: An International Review, 50(3), 436–454.

19.

Jain

R. K.

, & Agrawal

(2005). Indian and international perspectives on employee training practices: A trend. South Asian Journal of Management, 12(1), 79–100.

20.

Kirkpatrick

D. L.

(1959a). Techniques for evaluating training programs: Part 1-reactions. Journal of the ASTD, 13(11), 3–9.

21.

Kirkpatrick

D. L.

(1959b). Techniques for evaluating training programs: Part 2-learning. Journal of the ASTD, 13(12), 21–26.

22.

Kirkpatrick

D. L.

(1960a). Techniques for evaluating training programs: Part 3-behavior. Journal of the ASTD, 14(1), 13–18.

23.

Kirkpatrick

D. L.

(1960b). Techniques for evaluating training programs: Part 4-results. Journal of the ASTD, 14(2), 28–32.

24.

Kirkpatrick

D. L.

(1996). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler.

25.

Kozlowski

S. W. J.

, Gully

S. M.

, Brown

K. G.

, Salas

, Smith

E. M.

, & Nason

E. R.

(2001). Effects of training goals and goal orientation traits on multidimensional training outcomes and performance adaptability. Organizational Behavior and Human Decision Processes, 85(1), 1–31.

26.

Kraiger

, Ford

, & Salas

(1993). Application of cognitive, skill-based and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311–328.

27.

Laker

D. R.

, & Powell

J. L.

(2011). The differences between hard and soft skills and their relative impact on training transfer. Human Resource Development Quarterly, 22(1), 111–122.

28.

Lechner

(2000). An evaluation of public-sector-sponsored continuous vocational training programs in East Germany. Journal of Human Resources, 35(2), 347–375.

29.

Lee

S. H.

, & Pershing

J. A.

(2002). Dimensions and design criteria for developing training reaction evaluations. Human Resource Development International, 5(2), 175–197.

30.

Lim

D. H.

, & Morris

M. L.

(2006). Influence of trainee characteristics, instructional satisfaction, and organizational climate on perceived learning and training transfer. Human Resource Development Quarterly, 17(1), 85–115.

31.

Morgan

R. B.

, & Casper

W. J.

(2000). Examining the factor structure of participant reactions to training: A multidimensional approach. Human Resource Development Quarterly, 11(3), 301–317.

32.

Oblinger

(2003). Boomers, Gen Xers & millennials: Understanding the new students. Educause. Retrieved from www.educause.edu/ir/library/pdf/erm0342.pdf

33.

Olsen

J. H.

(1998). The evaluation and enhancement of training transfer. International Journal of Training and Development, 2(1), 61–75.

34.

Partridge

, & Hallam

(2006). Educating the millennial generation for evidence based information practice. Library Hi Tech, 24(3), 400–419.

35.

Payne

S. C.

, Youngcourt

S. S.

, & Beaubien

J. M.

(2007). A meta-analytic examination of the goal orientation nomological net. Journal of Applied Psychology, 92(1), 128.

36.

Pendry

L. F.

, Driscoll

D. M.

, & Field

S. C.

(2007). Diversity training: Putting theory into practice. Journal of Occupational and Organizational Psychology, 80(1), 27–50.

37.

Plant

R. A.

, & Ryan

R. J.

(1994). Who is evaluating training? Journal of European Industrial Training, 18(5), 27–30.

38.

Pont

(1996). Developing effective training skills. A practical guide to designing and delivering group training (2nd ed.). London: McGraw-Hill Book Company Europe.

39.

Punj

, & Stewart

D. W.

(1983). Cluster analysis in marketing research: Review and suggestions for application. Journal of Marketing Research, 20(2), 134–148.

40.

Roberson

, Kulik

C. T.

, & Pepper

M. B.

(2001). Designing effective diversity training: Influence of group composition and trainees experience. Journal of Organizational Behavior, 22(8), 871–885.

41.

Rudd

, Vohs

K. D.

, & Aaker

(2012). Awe expands people’s perception of time, alters decision making, and enhances well-being. Psychological Science, 23(10), 1130–1136.

42.

Russ-Eft

, & Preskill

(2001). Evaluation in organizations. Cambridge, MA: Perseus.

43.

Sandeen

(2008). Boomers, Xers, and Millennials: Who are they and what do they really want from continuing higher education? Continuing Higher Education Review, 72, 11–31.

44.

Sharma

(2014). Importance and performance of managerial training in Indian companies—An empirical study. Journal of Management Development, 33(2), 75–89.

45.

Shaw

, & Fairhurst

(2008). Engaging a new generation of graduates. Education + Training, 50(5), 366–378.

46.

Sitzmann

, Brown

K. G.

, Casper

W. J.

, Ely

, & Zimmerman

R. D.

(2008). A review and meta-analysis of the nomological network of trainee reactions. The Journal of Applied Psychology, 93(2), 280–295.

47.

Sugrue

, & Rivera

R. J.

(2005). State of the industry: ASTD’s annual review of trends in workplace learning and performance. Alexandria, VA: American Society of Training and Development.

48.

Tan

J. A.

, Hall

R. J.

, & Boyce

(2003). The role of employee reactions in predicting training effectiveness. Human Resource Development Quarterly, 14(4), 397–411.

49.

Tannenbaum

S. I.

, Mathieu

J. E.

, Salas

, & Cannon-Bowers

J. A.

(1991). Meeting trainees’ expectations: The influence of training fulfillment on the development of commitment, self-efficacy, and motivation. Journal of Applied Psychology, 76(6), 759–769.

50.

Van Buren

M. E.

, & Erskine

(2002). The 2002 ASTD state of the industry report. Alexandria, VA: American Society of Training and Development.

51.

Velada

, & Caetano

(2007). Training transfer: The mediating role of perception of learning. Journal of European Industrial Training, 31(4), 283–296.

52.

Yadapadithaya

P. S.

, & Stewart

(2003). Corporate training and development policies and practices: A cross-national study of India and Britain. International Journal of Training and Development, 7(2), 108–123.