Evaluating Training for New Government Officials: A Case Study Using the Success Case Method

Abstract

This study evaluates the training program for newly appointed government officials in Korea and presents the effectiveness of the success case method (SCM) technique for such an evaluation. The training program chosen as the case study is administered by the training institute for public officials of Korea. The SCM technique was used to evaluate this program and to draw implications that could be applied to the design of future training programs. A survey was conducted targeting 321 trainees who completed the program for newly appointed government officials, and 12 trainees were selected who showed a high level of training transfer measures for the qualitative examination. Through the analysis of semistructured interviews, the study deduced the factors that influenced effective training transfer. The results illustrate that the factors that lead to successful application of the training are 15 success factors categorized in the trainee’s individual characteristics, the training design, and the trainee’s work environment. This study provides implications that can be useful in the workplace, including the application of the SCM in evaluating training programs in the public sector, and the factors that should be considered for effective training transfer among newly appointed public officials.

Keywords

success case method training transfer training evaluation public sector training case study

Introduction

A large number of organizations have recognized that developing employee talent is the only way to maintain competitiveness, especially in environments of rapid and dramatic change, and are thereby expanding investment in human resource development (C. Lee, Lee, Lee, & Park, 2014). This emphasis on training does not exclude the public sector. As the public sector workplace continues to confront a range of challenges, organizations must commit to the strategic development of employees’ knowledge, skills, and abilities (Pynes, 2013). The discussion on the need for training has a long history. About 40 years ago, Mikesell, Wilson, and Lawther (1975) noted that training could improve many government activities and achieve better job performance in a reasonably short period of time.

One of the important issues in training and development is to determine the effects of training intervention. This means, it is critical to identify the effectiveness of the training program as it provides the key information for decision makers around further training investment (van Buuren & Edelenbos, 2013). Many scholars have discussed the training effect and how to evaluate it. For example, Kirkpatrick’s (1976) four-level evaluation has been used extensively in many training areas and several studies on public sector training evaluations utilize it as well (e.g., Bjornberg, DellCioppia, & Tanzer, 2002; Getha-Taylor, Fowles, Silvia, & Meritt, 2015; Olivero, Bane, & Kopelman, 1997).

One of the most important concepts in evaluating the effectiveness of a training program is training transfer, which is also a major concern in the Kirkpatrick model’s behavior level. Generally, training transfer is defined as the application of knowledge, skills, and attitudes learned from training on the job and subsequent maintenance of them over a certain period of time (Baldwin & Ford, 1988; Cheng & Ho, 2001). Successful training transfer has been recognized as the means to reduce the gap between employee training and actual performance (C. Lee et al., 2014). In a training program for public officials, the evaluation goal for training transfer is the same—to confirm the degree to which trainees apply effectively to their work after obtaining the knowledge and skills in the training program—and is considered of great importance, along with the need for a methodical approach and systematic analysis.

In this context, traditional training evaluation models, such as Kirkpatrick’s (1976), have been criticized because they only focus on the performance of the “training program” and not on various environmental factors impacting performance. Bates (2004) pointed out that the Kirkpatrick model excluded and simplified the various environmental factors surrounding the training program, resulting in popularity; however, it failed to systematically explore the various factors that affect the effectiveness of a training program. To address a general frustration with such traditional evaluation models, the success case method (SCM) was developed. SCM is a process for evaluating the business effect of training that is aligned with and addresses the organization’s strategy (Brinkerhoff, 2003). The SCM identifies the critical success and failure factors of training transfer for the overall analysis, and has been useful in qualitatively estimating and validating the effectiveness of many training programs (Choi & Lee, 2011).

Although the SCM technique has been applied to a wide range of areas, there are scant cases where it is used to evaluate training programs in public sector. One reason for the difficulty in applying SCM to public sector training programs was that SCM focuses on specific business outputs. However, recently, research has been conducted on educational officers applying the SCM process in the public sector in Korea (Lee, Choi, & Park, 2013). Research has also been conducted on U.S. health care officials (Medina et al., 2015) and on nonprofit social service organizations (Coryn, Schröter, & Hanssen, 2009).

This study utilizes the SCM technique to examine a case study of a specific training program for public servants. The study aims to evaluate various aspects of this training program and explore the factors that influence the effectiveness of it for newly appointed government officials in Korea. Through this research, the study attempts to increase the applicability of SCM to the evaluation of training programs for public servants and to suggest various ways that can improve training transfer.

Literature Review

Training Program Evaluation in the Public Sector

A training program evaluation is the process of assessing the value of the program, and this can be done by evaluating the program’s results, namely, the program’s performance, and by evaluating the program’s whole process that involves the program’s intention, content, and implementation, or by using both methods simultaneously. In the existing literature, the specific definition of “training program evaluation” may vary among researchers, but there is a shared concept of the training program evaluation as the process of identifying the program’s value, achievement of objective, and effectiveness.

The evaluation includes collecting information such as the human and material resources involved and analyzing the data needed for the training plan. In judging the usefulness of the collected information, the validity of the overall training plan can be examined, the improvement or elimination of the program explored, and the need for the quality improvement of lecturers or educational administrators assessed. In addition, the evaluation validates the legitimacy of the educational investment and promotes its usefulness. Thus, the significance of the training program evaluation can be understood more clearly through the need for the evaluation.

In looking at the public sector, Sims (1993) explained that the evaluation of the training program contributes to improving the effectiveness and efficiency of the program, increasing the reliability of the activities of the training officers, providing important information for the decision-making process, and increasing the commitment level of decision makers regarding training. Accordingly, numerous training institutes for public servants utilize various practices to assess the effectiveness of their training program (Bjornberg et al., 2002). For instance, Broward County in Florida used pre- and posttraining surveys targeting participants and their supervisors to conduct three-level behavior evaluations. However, the method that continues to be used most often in training program evaluations for public servants relies on the traditional Kirkpatrick’s four-level evaluation model; yet, this model only focuses on the quantitative aspects and leaves out various other factors that could influence the effectiveness of training.

The SCM

There is no doubt that Kirkpatrick’s model has made valuable contributions to training evaluation theory and practice. This model poses the need for training evaluation experts, intuitively presents the four results obtained through training, and simplifies the complexities involved in evaluating the training program (Bates, 2004). Nonetheless, this model has received much criticism in terms of insufficient evidence of causal relation and lack of consideration of contextual factors surrounding training programs. In particular, the Kirkpatrick model has been criticized as unsuitable because it excludes inquiry beyond the “training alone” and has no focus on the larger performance environment.

The SCM is a process for evaluating the business effect of training that is aligned with and fulfills the organization strategy (Brinkerhoff, 2003), and was developed to address frustration with more traditional evaluation approaches. The SCM, on the contrary, is intended to produce concrete evidence of the effects of training (or the lack of it) that senior managers and others find highly believable and compelling. Specifically, the evidence relates verifiable incidents of actual trainees who use their training in specific behaviors that can be convincingly shown to lead to worthwhile organizational results.

Moreover, SCM is based on the assumption that the evidence of training effects is always a function of the interaction of the training with other performance system factors. For example, “Whether their manager helped them prepare for the training and mutually set objectives”; “Whether the new commission structure was being faithfully applied in all sales transactions as per the new policy”; “Whether the training was provided within three months of new product introduction in the district” (Brinkerhoff & Dressler, 2003, p.8). The SCM seeks out and identifies such factors so that credit can be given and feedback provided to the parties involved.

The SCM approach involves two major steps. In the first, a brief survey is sent to a large representative sample of all the trainees who participated in the training. In essence, this survey asks one key question: “To what extent have you used your recent training in a way that you believe has made a significant difference to the business?” From the survey, small groups of exceptionally successful and unsuccessful trainees are identified. In the second step, each of these groups is examined more closely through in-depth interviews. The success case study produces two immediate results of in-depth stories of documented positive training outcomes and the knowledge of the factors that enhance or impede the effects of training on work-site results.

The SCM was originally designed for evaluating training programs in the private sector; however, it can contribute to the evaluation of public sector education programs also. Finding ways to increase the effectiveness of educational programs in the public sector has always been important. The SCM is well worth applying, as it can explore various factors that make training programs effective. How to define success is the key to the problem of applying public sector programs (Coryn et al., 2009). It is possible to collect successful cases and identify their reasons for success by defining what “success” is through the training program. This is because the SCM collects empirical cases of training transfer and performance (Brinkerhoff, 2005), and seeks to find the cause of such positive results from a systematic viewpoint.

Factors of training transfer

Transfer of training is mentioned as a concept that directly relates to reducing the gap between training and performance. There are various models for training transfer that reflect the different points of view of the scholars. Baldwin and Ford’s (1988) transfer process model was a primary model on early research on transfer, acknowledging the factors of trainees’ personal characteristics, training design, and work environment as the three categories of training input factors that affect the performance and transfer of training. This research is recognized as providing the framework distinguishing types of factors that influence the transfer of training. Moreover, this model has had a continuous effect on subsequent studies, and thus, there has been research on the effects of variables that can be subsumed under these categories (Burke & Hutchins, 2007).

Gegenfurtner, Veermans, Festner, and Gruber (2009) presented an integrative model of motivation on the training transfer. This model describes the transfer of training in terms of individual, training-related, and organizational factors, illustrating that the motivation to transfer, with a mediating effect on all variables, has an effect on the transfer of training. They made a distinction between pre-, mid-, and posttraining periods when looking into factors influencing training transfer.

Choi and Lee (2011) proposed an integrative model of training transfer for outstanding performers with a high level of training transfer, explaining the influence of key success factors on the whole process of learning and the transfer of training through in-depth interviews with such performers. This model illustrates that such performers had high expectations on improving their performance through the training before they attended the program, and acknowledged high relativity between the training content and their work. Such individual and organizational characteristics influenced their motivation to learn, which in turn influenced their training transfer motivation, leading to the prediction of successful training transfer.

Method

Case Study Description

The Training Institute is the representative organization that trains public servants in Korea and takes an essential role in sharing the direction of government operations, raising the level of the nation’s talented individuals. This study examines the training program for newly appointed government officials, they are generally in charge of policy plans, policy implementation, and management tasks within the administrative organization. Specifically, the trainees take on the following roles: an information manager who collects and analyzes the information necessary for policy planning; a policy planner who identifies the existing policy’s problems and needed improvements, plans policy, and writes reports; and of a mediator who is an intermediary manager of an organization and conducts the effective cooperative relations and communication between upper and lower management groups.

The newly appointed government official program is a long-term training program (27 weeks, 1,048 hr) that aims to establish a desirable public servant image, develop creative job competence, and grow global competence in the era of globalization. More specifically, the training is generally composed of six modules: development of the value of public service, development of job competence, development of manager competence, development of global competence, experience in the field of public administration, and development of basic fundamentals. As this study attempts to evaluate training transfer, it selected the module of development of job competence taking into account the fact that it must be possible to estimate whether what is learned is used at work. The training hours for the job competence module are 27.9% (292 hr) of the total training hours, and it consists of critical tasks of the government officials including policy planning and implementation and the knowledge and skills needed to accomplish the monitoring role. More specifically, it consists of policy planning practice, research on policy cases, and fundamental job training. It also includes highly applicable content that allows the trainees to work in small groups to practice writing legislation and budget proposals based on the actual policy planning process. It also enables the trainees to learn in detail the fundamental job competency that is essential for accomplishing responsibilities such as human resource affairs, security, administrative procedures, and national accounts, and to write different reports on the basis of what they learn.

Data Collection Procedures

The data collection process was conducted based on the steps suggested by Brinkerhoff (2005), illustrated in Figure 1. First, among the 19 trainees in the development of job competence module, the subjects that could estimate the training transfer were selected. Second, the training effect model for the subjects and then, third, the training transfer measurement tool were developed. The fourth step was the estimation of the level of training transfer and, the fifth, the in-depth interviews with those trainees with outstanding levels of training transfer. The final step was the deduction and analysis of the critical success factors of training transfer.

Figure 1.

Illustration of SCM steps.

Selection of subjects

In selecting the subjects that could estimate the training transfer, criteria were designed based on classification standards of evaluation steps as suggested in existing literature (Keller, Morrison, & Ross, 1994; Phillips & Stone, 2002). The training transfer—namely, the evaluation of work applicability (Level 3 evaluation)—was categorized into four classification standards: value of work application, degree of training course development that takes into account work application, degree of support acquisition of work applicability evaluation, and the feasibility of measurement of work applicability. Subsequently, the study identified each subject’s learning objective and core learning materials and categorized them according to the classification standards of evaluation steps. The draft of the classification of each subject’s evaluation steps was then revised and supplemented from interviews with professors who have lectured on the subject. This was ultimately confirmed through focus group interviews (FGI), in which administrators, professors, and decision makers participated. As a result, five subjects were selected as relevant for evaluating work applicability: policy planning practice, research on policy cases, and practice writing reports by type, legal work, and budget work.

Development of training effect model

Three procedures were implemented to develop the training effect model. The first involved reviewing the learning objective and core learning materials established in the training planning process; the draft of the training effect model was then deduced from the textbook content. The second involved interviewing the administrator and professor of the training course and revising the draft of the training effect model appropriately. This was to confirm that the content of the training effect model was aligned with the learning material. After the interview, (a) any differences between the textbook content and the budget work lectures were amended, and (b) parts of the critical actions and key results that were inappropriate for the work-site were revised. Last, the third procedure involved three PhD students, who majored in HRD (Human Resource Development), verifying the supplemented training effect model. This expert verification was implemented to check whether the training effect model was developed appropriately to facilitate the measurement of training transfer and whether it was developed in accordance with the procedures of the SCM. After the examination by the experts, (a) the verbs for critical actions were revised, and (b) any ambiguous expressions of each factor were deleted and revised from the training effect model draft. Ultimately, the training effect model was confirmed on the basis of the interviews with the course administrator and professor and the examination by the experts (see Table 1).

Table 1.

Training Effect Model: An example of policy planning practice.

Key knowledge and skills	Critical action	Key results
Concept and steps of policy planning Comprehension of the policy problem and method of cause analysis Participants of policy course and participation pattern Major steps of policy work Concept and steps of policy quality management Types of report by policy work and its necessity Steps for writing reports and their content Concept of legislation and the basic principles of drafting Legislative procedures of government and national assembly Methods of enactment, amendment, and rescission of legislation	To progress in accordance with the procedure of the policy plan To systematize the report focusing on situation, problem, and improvements when writing policy planning reports To identify whether the improvements resolve the real-life problems either partially or wholly To understand the demands of major stakeholders and participants, and prepare for expected questions when planning a policy To implement policy work by distinguishing the steps To perform based on the checklist for policy quality management To forecast and prepare for the reports necessary for each task To find similar reports and refer to them when writing reports To prepare and utilize the templates of reports that have been often written for the relevant tasks To write the legislation draft in accordance with either whole or partial amendment forms of the legislation draft To review the legislative progress To suggest the particular business’s requirements, details of calculation, reason for support, and necessity of support when writing the budget report for the particular business To write the reference that proves the necessity of business support when writing the budget report for the particular business	Production of systematic reports that follow the policy planning steps Improvement of solvability of real-life problem of established policy Increase in the satisfaction of major stakeholders of policy Systematic accomplishment of policy work Improvement of writing skills for different types of reports Production of accurate amendment legislation Production of budget proposal for the particular business based on details of calculation, reason for support, and the necessity of support Accurate production of budget proposal’s supporting evidence for the particular business

Development of measurement tool of training transfer

The measurement training transfer tool was developed by categorizing general transfer items and each subject’s transfer items. The general transfer questions were developed on the basis of existing literature (Holton, Bates, Seyler, & Carvalho, 1997). For the measurement of each subject’s transfer factors, the draft of a behavior index of work applicability was developed examining the subjects’ learning objectives and core learning materials. The behavior index was then revised and supplemented through the FGI with the professors of each subject. Through this process, four general transfer items and five training transfer measurement areas per subject were developed. For example, these included statements such as “I actually applied knowledge and skills obtained from the training program to the job” and “When I plan a policy, I follow the four policy planning steps.” The results of analyzing the reliability of the developed measurement tool illustrated that the Cronbach’s alpha was statistically significant at .879 for the general transfer items, and for each subject’s transfer items, the Cronbach’s alpha was .785 for policy planning practice, .769 for research on policy cases, .724 for writing different types of reports, .767 for legal work, and .846 for budget work, all of which were statistically significant. All items were measured on a 5-point Likert-type scale.

Estimation of training transfer

The online survey to estimate the training transfer was conducted over 10 days and was sent to 321 trainees in the newly appointed Government Officials Training Program. The estimation of the training transfer was assessed about 12 months after the completion of the program. By this time, the trainees had been assigned to their jobs and were at work after having completed 6 months of on-the-job training (OJT). Of the 321 trainees, 198 responded, with a final count of 139 valid responses used in the analysis (valid response rate: 43.3%). In terms of the general characteristics of the respondents, 59.7% were male, 49.6% were general administration workers, 53.2% were between the ages of 25 and 29, and 75.5% worked at the central department, a relatively high percentage. In addition, these ratios were similar with those for the total population of the training program for the newly appointed government officials, thereby confirming that the study sample had a similar component ratio with the program’s total population.

In-depth interviews with trainees with a high degree of training transfer

Based on the results of the survey, the study selected the outstanding trainees who represented a high level of training transfer. The outstanding trainees were categorized as the top 10% according to Pareto’s law, which is the SCM’s basic assumption. These 14 outstanding trainees, who represented 10% of the 139 valid respondents, were then chosen as the appropriate targets for the first interviews. Moreover, looking at the ratio of participants in the training program for each job among the newly appointed government officials, the outstanding trainees were from the following departments: three in the finance and economy department, seven in the general administration department, three in the technology department, and one in the international trade department. Subsequently, based on the training transfer survey results, only the respondents with an average level of training transfer above 4.0 were selected. This step eliminated two trainees from the general administration department, reducing the number of trainees from that department to five. The numbers of trainees from the other departments remained as stated above (see Table 2).

Table 2.

Characteristics of the Study Participants.

Classification	Sex	Department	Number of working months	Age	Average of general transfer	Average of job transfer
A	Female	Finance and economy	5	25	4.14	4.67
B	Female	General administration	5	31	4.21	4.00
C	Female	General administration	4	26	4.00	4.00
D	Male	Technology	4	32	4.25	5.00
E	Male	Technology	6	35	4.69	4.50
F	Male	Finance and economy	4	25	4.36	4.33
G	Female	General administration	3	29	4.00	4.00
H	Male	Finance and economy	4	28	4.86	5.00
I	Male	Technology	4	31	4.00	4.00
J	Female	International trade	3	24	4.27	3.67
K	Male	General administration	5	26	4.00	4.00
L	Male	General administration	4	31	5.00	5.00

To conduct in-depth interviews, two researchers visited the outstanding trainees in person and conducted in-depth interviews lasting about 2 hr each. The content from the in-depth interviews was recorded with the prior consent from the interviewees, and the records were transcribed to increase the accuracy of the content. The accurate recording is a method used to maintain the qualitative research validity (Wolcott, 1990). This study not only described the interviews but also included all the detailed information about the interview progress, including the interview date, interview hours, interview place, and interviewees. The semistructured interview questionnaire developed for the study was sent 7 days ahead of the interview for the interviewee to prepare responses in advance. Moreover, to conduct the interview under natural settings, the research objective and information about the interview were first explained, and the interviewer began with open-ended questions about the interviewee’s work experience (see Table 3).

Table 3.

Semistructured Interview Questionnaire for SCM.

Classification	Key questions
Success case applying training program into practice	Could you tell us a specific example of applying what you learned through the training program? • When and under what circumstances? • How did you apply? • How has that case helped your teams and organizations achieve their goals? What do you think is the ultimate value of participating in the training program? • Why do you think the result is important? • How valuable is the result?
Influencing factors for training transfer
Individual characteristics	What are some of the characteristics you have, such as your personality, that help you apply what you have learned in the training program to your work? What are some of the environmental characteristics of your environment, such as your family environment, life history, and so on, which help you apply what you have learned in the training program to your work?
Training design	Which of the lessons you learned in the training program could relate to your current work? Which of the lessons you learned in the training program could apply directly to your current work? What are some of the activities offered in the training program that helped you apply them in your current work?
Work environment	At the time of your participation in the training program, did your supervisor, colleagues, and organization help you participate in the course? How did your supervisor, colleagues, and organization help you apply what you learned in the training program to your work? Did your organization provide you with the opportunity or situation to apply what you learned in the training program to your work?
Other factors	What other factors have been the driving forces in applying what you have learned in the training program to your work?

Note. SCM = success case method.

Data Analysis

To analyze the level of training transfer, this study calculated the descriptive statistics that include minimum value, maximum value, average, and standard deviation. The results of the analysis were used in evaluating the level of training transfer and in selecting outstanding trainees. Each in-depth interview that targeted the outstanding trainees was analyzed through content analysis. Frankfort-Nachmias and Nachmias (2002) defined content analysis as the method of systematically and objectively identifying and inferring the detailed characteristics of the messages and explained that objectivity could be ensured by conducting the analysis according to explicit rules where many researchers participate in the analysis process. The contents of the recorded interviews were documented according to the guideline suggested by Hycner (1985); when the researchers’ interpretations differed, the content was reanalyzed and appropriate meanings added through further detailed verbal and nonverbal investigations to reveal the interviewee’s intention, the interview content was then categorized into major themes through a meeting of the minds of the researchers, and the literature for each categorized theme was examined and organized. The contents of the recorded interviews are first documented according to the procedure mentioned above, categorized according to each factor, and recategorized within each factor to systematize the interview content. The study analyzed the statements of the interviewees, and those that were repeatedly emphasized or mentioned were noted separately to interpret the interviewee’s intention. When one sentence, without being finished, occasionally digressed from the point and continued with a different idea, a separate statement was written for the analysis purpose. The data first analyzed by one researcher were reconfirmed by coresearchers who conducted the interviews together to secure the validity.

Results

Levels of Training Transfer

The results from the estimation of the training transfer for all the trainees (N = 139) who participated in the program reveal an average training transfer of 3.52. In terms of each subject, the average of “writing different types of report’” was 4.09, which showed the highest level of transfer. The averages for “policy planning practice” and “legal work” were 3.94 and 3.67, respectively (see Table 4).

Table 4.

Average Values for Training Transfer by Subject.

Classification	Minimum value	Maximum value	Average	SD
General transfer	2.00	5.00	3.52	0.706
Transfer for each subject
Writing different types of report	2.00	5.00	4.09	0.662
Policy planning practice	1.00	5.00	3.94	0.780
Legal work	2.00	5.00	3.67	0.746
Budget work	1.67	5.00	3.53	0.755
Research on policy case	1.67	5.00	3.40	0.698

Critical Success Factors for Training Transfer

The study investigated the factors that influence training transfer and facilitate the process of training transfer through interviews with the outstanding trainees who successfully apply the skills they learn to their work. The results illustrated that the critical success factors of training transfer can be categorized mainly into individual characteristics, training design, and work environment (see Table 5).

Table 5.

Critical Success Factors of Training Transfer.

Category	Subcategory	Concept
Trainee characteristics	Trainee’s learning willpower	• Engaged in the training with the attitude to enjoy • Attempted to concentrate on lecture contents
	Adding meaning to training content	• Tried to understand one’s work context such as governmental philosophy • Applied additional interpretation of the learning to one’s work background or the overall context • Engaged in the training with the thought that writing a report is an important part of the work
	Long-term perspective toward the effect of training	• Thought that what they learn would be helpful some time in future despite no direct connection • Experience from overseas training would be helpful for policy planning in future • Considered the fieldwork such as country pilgrimage to be helpful in the long term
	Active participation in task activities	• Active participation in projects within a group helped in improving practical skills • Various experiences could be obtained by actively participating in all tasks
	Trainee’s transfer willpower	• Attempted to begin with searching or referring to rules when performing work after completion of training • Attempted to utilize what they learned, whatever they obtained from the training • Attempted to reduce the work hours by using training content • Attempted to put the procedures learned from the training into practice
	Utilization of learning materials	• Searched the learning materials first when establishing a new legislative system • Consistently used the materials from policy planning or information system course
Training design	High work relevance of training contents	• Frequently utilized policy planning or report writing because it is highly related to work • Writing policy report about economy, highly related to current work, was helpful • Frequently applied writing specific budget proposals as was highly work relevant • Frequently used the content from policy planning practice when establishing the report’s table of contents
Category	Subcategory	Concept
	Suggestion of real-life example	• The experience and real-life examples from work-site were helpful• Frequently used the tips from director’s lecture, including the manners of supervisors and the reporting method• The outstanding report examples were helpful when performing work
	Education through practice	• The process of facing difficulty during the policy planning practice was used when performing work• Frequent utilization of basic report template to write a report was helpful• Role-playing was helpful in mediating conflicts• Practice of writing reports including research on policy cases was frequently utilized
	Voluntary task execution	• The experience of direct participation to complete a report improved the specific skills• Selecting the regions to visit and subjects for country pilgrimage increased the effects of training• Overall planning, liaising, and meeting through visiting the actual institutes and conducting interviews during the overseas training were very helpful• The content that was studied through direct research per project remained relevant for a long time
	Experience in overall process	• Experience of the whole process of actual overseas business trip through overseas training was helpful• A series of processes including establishing, implementing, and executing plan through group projects was helpful• Learning the work process during the local government training was helpful• Learning the procedures of national assembly and budget work was helpful
	Interaction and networking among trainees	• Improved the teamwork through country pilgrimage• Helped mutually after the program through interactions among trainees• Cooperated based on network formed among colleagues from training institute• Utilized the network obtained from other regions
Work environment	Opportunity to utilize at current work	• There are many opportunities to utilize the learning content because there are numerous works involving training Hangul/ PowerPoint(software) or searching overseas cases• There are many opportunities to write a report because he or she is performing planning work• The trainee was able to utilize relevant learning content because the trainee currently works as an intermediary manager• The work involves writing a report almost every day• Skills from overseas training are frequently used because the current work involves contacting departments, suggesting concept notes, forming consultative groups, and writing reports• Understanding of administrative procedures was helpful in establishing different standards because the trainee is currently conducting investigation work• Manners and greetings learned from training are frequently used because the trainee performs work that involves meeting many different people• It could only be utilized in writing a report, meeting people, and holding a meeting• Training content was well utilized when systematic planning work was given
	Change-oriented work characteristics	• It was easy to apply the content because the work changes rapidly• It is possible to conduct numerous attempts because many new ideas are planned and suggested• Due to work characteristics, we have no choice but to continue studying the latest technology
	Learning-oriented culture	• Supervisors and colleagues are open to learning and applying new skills• Supervisors think that it is important to acquire new skills• Supervisors complement when new skills are applied• There are many academic groups within the organization, and training is conducted regularly

Individual characteristics

The individual characteristics that affected the training transfer for the outstanding trainees were identified as follows: the trainee’s learning willpower, adding meaning to the training material content, a long-term perspective regarding the effects of training, active participation in task activities, the trainee’s transfer willpower, and on the job utilization of learning materials.

First, these trainees reported that an equally high level of motivation to learn facilitated the training transfer. Various forms of effort for learning were recalled. Among them, these trainees had the will to directly study the materials that they were given.

People often fall asleep while taking courses because numerous classes are conducted in the form of lecture. I was the one who did not fall asleep because I was accepted through passing the examination after working as a salaryman for about 4 years. One of the things that I desired the most was to listen to the information that is not relevant to my field, such as this course. So I often make an effort to concentrate when I listen. (Research Participant E)

Second, these trainees motivated themselves by adding further meaning to the training content they studied. They could have more achieving and enthusiastic attitude by thinking the training content is significant, creating new association among other training materials, and understanding the training experience in a long-term perspective.

I eventually thought that it is important to understand the background because it is a smaller field than I expected. Nevertheless, it will be meaningless if I miss these learning materials that are highly related to the governmental philosophy. Hence, I thought that the education related to governmental philosophy was important. (Research Participant I)

Third, the outstanding trainees took a long-term perspective toward the training content to motivate themselves. In particular, they had a positive attitude, thinking that what they learned would help them in their public careers sometime in the future even though it could not be immediately applied to their work.

In terms of the country pilgrimage, it emphasized the numerous opportunities to meet the citizens. What I interviewed and heard would not be immediately helpful, but I think that they would be helpful in the long term. (Research Participant L)

Fourth, the outstanding trainees attempted to actively participate in the group activities to maximize the training effect. Active participation in various tasks appeared in the form of taking an important role in a group and trying to obtain various experiences through task activities.

Every group has different task activities, leading to cause a situation in which certain people take on the majority of tasks. I tried to be active in participating in every given task. It was personally tiring, but I could obtain various experience by participating in one thing and another. (Research Participant F)

The trainee’s learning willpower, further interpretation of the training content, long-term perspective toward the effects of the training, and active participation in the task activities all relate to the trainee’s motivation to learn. Motivation to learn refers to the persistence and concentration of the effort to implement improved activities based on what they learned before, during, and after the training (Tannenbaum & Yukl, 1992). Of the numerous studies that reveal the factors that affect training transfer, this, in particular, has been considered as the most important variable.

Thus, the outstanding trainees generally attempted to apply what they learned or to continuously utilize the learning materials from the training. This then relates to the trainees’ motivation to transfer the training. Motivation to transfer is defined as the trainee’s intended effort to use the knowledge and skills they learned from the training in real-life work situations (Noe, 1986; Noe & Schmitt, 1986).

Before taking the program for the newly appointed government officials, I did not try to search for laws or did not even have the concept of referring to laws. After taking the course for 6 months, I began to search the relevant laws or confirm with guidelines. I feel that I am trying to apply what I learned in this case. (Research Participant I)

Making the connection with the government project when establishing a policy or having the opportunity to listen to the expert’s opinion was the steps that I attempted to take in order to put what I learned from the government project or policy planning course into practice. (Research Participant C)

I am still using the learning materials obtained from the policy planning course or information system course. I have not thrown them away, and I think they are really useful. (Research Participant L)

The critical success factors for training transfer in terms of the individual characteristics identified in the in-depth interviews could be summarized as motivation to learn and motivation to transfer. Motivation to learn and motivation to transfer have often been mentioned in the literature as the individual characteristic variables that affect training transfer. Moreover, they have been verified to be the precedence factors of training transfer in empirical studies (e.g., Axtell, Maitlis, & Yearta, 1997; Chiaburu & Marinova, 2005). An existing study (C. Lee, Choi, & Park, 2013) that investigates Korean teachers to reveal training success factors also demonstrated that the trainees needed the motivation to learn the knowledge and skills provided through the training as well as the motivation to apply them. Specifically, they revealed that the trainee’s interest in the training content and his or her passion toward learning affected the motivation to transfer in the future.

This study also illustrated that the participants who responded that they had a high motivation to learn generally also exhibited the will to apply what they learned. This corresponds with the results from another study (Chiaburu & Marinova, 2005) that the motivation to learn will affect the training transfer through the motivation to transfer. This study was additionally significant in that it revealed that the internal process of the trainee inspired the motivation to learn. In particular, it illustrated that the trainees motivated themselves by further interpreting the training content and having a positive long-term expectation of the effects of the training.

Training design

The training design factors that affect the training transfer among the outstanding trainees were identified as follows: the high work relevance of the training content, suggestion of a real-life example, voluntary task execution, experience in overall process, education through practice, and interaction and networking with other trainees.

First, these trainees reported that the level of training transfer was high for the subjects where the training content was highly related to their jobs. They explained that the content from the real-life examples that they saw and heard during the training was helpful for them in their work.

Despite the difference depending on departments assigned to trainees, writing and submitting the policy plan or individual reports practically had the highest work relevance for government officials since they generally write numerous reports. So I thought that it was very helpful. (Research Participant A)

Maybe it is because I saw the same template when I was practicing policy planning at the training institute that it did not seem unfamiliar. It was definitely harder when I was learning, but I thought that it was a good experience. When I wrote the previous year’s business performance report, I often applied the method I learned to specifically suggest the support basis and necessity of support for writing the budget proposal. (Research Participant J)

I thought that the real-life examples from the lecturers who came from the work-site were most helpful for me. I thought it was most effective to hear real-life examples such as how the work is progressed at the work-site and what they experience. (Research Participant G)

The outstanding trainees reported that education through practice was also very helpful. In particular, the opportunity to experience the tasks or situations that could arise at work led to training transfer when the similar situation did arise at work.

It is actually really important to cooperate and work as a team at work. For instance, programs such as role playing and mediating conflicts were helpful for competence education . . . Ultimately, it is the teamwork among which trainees often had to accommodate and mediate between different opinions, and this experience (through practice) was helpful. (Research Participant F)

At the same time, the outstanding trainees reported that they could accumulate the skills they needed at work through the experience of completing the projects assigned to each training group and that they could often use these afterward during the performance of similar work. Furthermore, they explained that the experience of exposure to the process, including establishing, implementing, and executing the plan or the comprehension of work procedures, was helpful in accomplishing their jobs.

The direct experience of accomplishing the project was most helpful for me. When the subject was given as a project, the entire group autonomously conducted the various tasks ranging from researching data to contacting institutes . . . There were more things we learned through conducting research by ourselves rather than through the cramming. (Research Participant H)

Our general tasks involved getting the legal or budget consideration, and the fact that I learned the procedure through national assembly work or budget work course was very helpful when I was allocated this at work. Here, no one tells you how the procedure works from the beginning. (Research Participant J)

Moreover, the fact that the training was designed to establish the interaction and network among trainees was said to be helpful in terms of mutual support when they performed their work.

I think that the biggest benefit I gained during the training was my colleagues at each department. Whenever I have a question or a task to cooperate with the ministry of strategy and finance, I obtain better cooperation when I first look for my colleague. So my reasoning is based on this. (Research Participant K)

The training design factors that facilitate the training transfer are significant in the fact that the HRD managers can be directly involved. With regard to this, existing studies have also revealed the training design factors that facilitate training transfer. As in this study’s results, the earlier literature suggests that the content relevance is the important factor that facilitates the training transfer (Axtell et al., 1997; Burke & Hutchins, 2007; Yamnill & McLean, 2005), along with the appropriate practice and feedback (Lee & Kahnweiler, 2000; Salas, Rozell, Mullen, & Driskell, 1999), and active learning (Burke & Hutchins, 2007; Silberman & Auerbach, 2006). Moreover, Burke and Hutchins(2008) indicated that ‘coaching and opportunities to practice’, ‘use of interactive activities to encourage participation’, and ‘making the content relevant to actual job duties’ were the most frequently reported strategies for fostering training transfer through analyzing the best practices reported by training professionals.

This study reports training design factors not observed in earlier literature; they are the opportunity to experience the overall process of work and the ability to establish a network with other trainees. These additional factors could have appeared due to the job characteristics of the government officials, specifically, applying similar processes to various projects, as well as the specific training characteristics of this program such as its length (6 months).

Work environment

The work environment factors that affect the training transfer for the outstanding trainees were identified as follows: the opportunity to utilize the training in relevant work, change-oriented work characteristics, and a learning-oriented culture.

First, the outstanding trainees generally reported that the transfer was facilitated as they received more opportunities to apply the training content at work. This could be similarly interpreted in terms of the degree of performance depending on the connection between the training content and the work. In other words, when there is a greater opportunity to apply what they learn from the training to real-life work, they exert more effort on the transfer.

I write one report in two days on average. I mean, I write a report almost every day . . . I had an experience of writing the four different types of reports at the training institute. Without this experience, I would have been confused about what to do on my arrival. It is because I did not know what to do after opening Hangul, but this experience was very helpful. (Research Participant E)

The overseas training experience was extremely helpful for me. In my department, I have to make direct contact with foreign organizations or departments in order to form a consultative group, and I have to write reports after business trips. Hence, the overseas training was extremely helpful, more than other subjects. (Research Participant F)

In addition, the trainees explained that it was easy to apply the training content to work when they took into account characteristics of their work that were sensitive to change. Also, they emphasized that not only work characteristics but also supervisors or colleagues within the organization generally encouraged learning and were supportive of adapting to change.

My work is related to international cooperation, of which the trend changes rapidly, and I have to learn new information in order to keep abreast of the trend. So I think our department is positive towards applying the learning materials to work. Also, since I am newly appointed, they want something fresh from me . . . (Research Participant A)

Due to the job characteristics, we have no choice but to continue studying the latest technology. So the center manager and the section chief think that learning and acquiring new technology is very important, and they complement more when we bring new information and apply to our center. (Research Participant E)

We hold seminars internally and autonomously once per week or two to three times per month. Employees write a report or PowerPoint about what they felt and learn from the work-site, and they lecture and discuss in front of all employees. (Our supervisors) encourage all of us to participate and be well-informed. (Research Participant H)

The critical success factors of training transfer related to work environment characteristics from this study seem to be generally identical to those from existing literature. Numerous studies have investigated the direct effect of work environment characteristics, which is the environmental factor that facilitates training transfer. Specifically, the studies about the organization’s transfer climate (Kontoghiorghes, 2001; Lim & Morris, 2006) explain that the support from the organization, manager, or coworkers has a direct effect on the effectiveness of the training. Baldwin and Ford (1988) indicated that the manager’s support and the opportunity to apply the training content to work are the important factors that support the trainees in maintaining their skills.

Moreover, researchers have consistently illustrated that there is a limit to positive training transfer when trainees are not provided with the opportunity to apply their new skills at work (Brinkerhoff & Montesino, 1995; Gaudine & Saks, 2004; Lim & Morris, 2006). In particular, a lack of opportunity to use the training materials from the program was considered the biggest obstacle to training transfer (Lim & Johnson, 2002). This study had some respondents who said they did not utilize the skills learned from the training on certain subjects because they did not have any opportunity for utilization. The lack of opportunity to use the training content at work due to reasons such as lack of time or a supervisor’s unilateral decision appears to be an important factor that hinders training transfer.

Discussion and Conclusion

The purpose of this study can be classified into two main objectives. First, it aims to represent the effectiveness of the SCM by using this methodology to evaluate the effects and effectiveness of a case study training program in Korea in which training for the public sector was analyzed using the SCM. Second, it aims to identify the factors that facilitate the effectiveness of training on the basis of the training transfer factors analyzed through the SCM. To accomplish this goal, the critical success factors of training transfer were derived by interviewing outstanding trainees from the newly appointed Government Officials Training Program of Korea after following and reviewing Brinkerhoff’s SCM procedure. The findings of this study are as follows.

First, this study aggregated the earlier literature and identified the suggested procedures to implement the SCM. More specifically, the procedures of SCM are as follows: the selection of subjects, the development of the training effect model, the development of the measurement tool for training transfer, the estimation of the level of training transfer, in-depth interviews with the trainees with the highest degree of training transfer (outstanding trainees), and the deduction of critical success and failure factors (the bottom 10% training scores) of training transfer. These six procedures suggested in this study can provide a specific guideline for evaluating a training program using SCM in the future.

Second, this study verified the critical success factors—largely classified into individual characteristics, training design, and work environment—that facilitate the training transfer. From the perspective of individual characteristics, the trainee’s high motivation to learn and motivation to transfer appeared to be the individual characteristics that facilitate the work-site application of the training. The training design factors that facilitate the transfer included training content that is highly relevant to the work, the real-life examples, opportunity to voluntarily execute tasks, education through practice, and active interaction and networking with other trainees. This study identified three work environment factors, which include whether the work that trainees perform includes change-oriented characteristics, whether the department culture of the current work is learning-oriented, and whether the opportunity to apply the training content to work is provided. The critical success factors of training transfer are significant in that they provide evidence of the relationship between the training transfer and antecedent training. This then provides various ideas to facilitate training transfer in the future.

The study began with the following discussion on utilizing the procedures of the SCM and its effectiveness. First, it is most important to define “success” in the beginning of the SCM application on the evaluation of training programs in public sector. In particular, the success of public sector training programs, unlike private companies, is often not quantifiable. Therefore, it is necessary to specify “success” and utilize the phase of development training effect model in the SCM process.

Second, assuming that the aim of the SCM is to estimate the training transfer, as it is in this study, the possible estimate of the training transfer in the first step (selection of subjects) of the SCM must be considered. For instance, questions as to whether the subject is designed such that its content can be applied to work, whether the training content will be utilized in actual work, whether there is enough budget to evaluate the subject’s work applicability, and whether these evaluations are valuable would all be the practical considerations.

Third, while existing evaluation methods of training programs have attempted to obtain the information from average or mid-level trainee success by relying on typical quantitative methods, the SCM is different in that it aims to obtain the implications through the successful stories of outstanding trainees who complete the training program. Hence, it is very important to establish the criteria of how to select these outstanding trainees, as well as the decision-making process for this. After developing a measurement tool for training transfer, this study surveyed trainees to estimate their training transfer and then selected the top 10% of the respondents with the highest training transfer grades as the first in-depth interviewees. Along with the training transfer estimate, a consideration as to whether there is a “successful case” applying the actual training content to work is also helpful in selecting the outstanding trainees to obtain specific success stories.

On the contrary, it is necessary to consider applying the following discussions in training programs in the public sector taking into account the influencing factors of training transfer found in this study. First, their effects, motivation to learn, and motivation to transfer on training transfer have been suggested by other studies (e.g., Axtell et al., 1997; Chiaburu & Marinova, 2005). However, this study presents specific examples of these effects. In particular, the outstanding trainees reported that to motivate themselves to learn, they pursue activities such as further interpreting the meaning of the training content and considering the effects of the training over the long term. Hence, officially including the time for such activities as part of future training programs would be an alternative to facilitate this type of work application.

Second, this study reports that active involvement of trainees could be a critical success factor in the transfer of training. Considering the lack of empirical studies on active learning and training transfer (Burke & Hutchins, 2007), this study is significant in that it reveals the training design factors that lead to trainee active participation. The implication is that these variables should be consistently reflected when designing future training programs. Moreover, these variables also need to be further validated by additional quantitative studies in the future.

Third, this study did report training design factors that have not been observed in earlier literature: the opportunity to experience the overall process of work and the activity of establishing the network among trainees. These new factors could have appeared in this study due to the job characteristics of the government officials, the application of similar processes to various projects, and to the characteristics of the training program, specifically its long term (6 months). This implies that the training program (the target employees and the content) and the characteristics of the training process could also affect the factors that facilitate the training transfer. This could be a topic for additional research validation in the future.

Finally, the factors that were frequently reported to facilitate training transfer were the opportunity for work application and the opportunities given by a supervisor or an organization. To facilitate the training transfer in the future, trainees who complete the coursework could notify the department head of what they learned, and the current work department could attract more attention by using a survey or interview to collect information about additional knowledge and skills necessary for newly appointed government officials.

This study is significant in that it provides a specific case application of the SCM technique for the evaluation of a training program in the public sector in Korea. Specifically, this study provides more detailed procedures and examples that can be utilized in the training in the public sector than those provided earlier by Brinkerhoff (2003). Through these procedures, qualitative analysis was conducted to deduce various factors that could facilitate training transfer. Most of the critical success factors found in this study were consistent with previous studies; however, some were derived from the inherent characteristics of the training program. Therefore, rather than generalizing the critical success factors found in this study, it will be necessary to focus on how to derive critical success factors. In addition, the SCM procedure suggested here must be validated by expanding its implementation to various targets and programs and supplemented to consider the other study limitations in the future.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This research was supported by National Human Resource Development Institute under the Ministry of Personnel Management in Korea.

Author Biographies

Chan Lee, PhD, is an associate professor and academic program chair of Vocational Education and Workforce Development at Seoul National University in Korea. His main research interests are in strategic human resource development, social learning, entrepreneurship, coaching and leadership development, performance management, job analysis, and structured on-the-job training(S-OJT).

Dongwon Jeon, PhD, is a vice-president at Do-How Consulting Corporation in Korea. His research areas include Informal learning (70:20:10 model), strategic HRD, performance evaluation, competency modeling, and development of competency-based curriculum.

Wooseok Kim is a senior consultant at Do-How Consulting Corporation. His research areas include Job satisfaction, performance analysis, and Informal learning (70:20:10 model).

Jaeeun Lee, PhD, is an assistant professor at Institute for Innovative Education of Konkuk University in Korea. She interests on the researches focused on evaluation of training effectiveness, relationship between work experience and expertise, and employees’ career development and success.

References

Axtell

C. M.

Maitlis

Yearta

S. K.

(1997). Predicting immediate and longer-term transfer of training. Personnel Review, 26, 201-213.

Baldwin

T. T.

Ford

J. K.

(1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63-105.

Bates

(2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27, 341-347.

Bjornberg

DellCioppia

Tanzer

(2002). Training and development: Best practices. Public Personnel Management, 31, 507-516.

Brinkerhoff

R. O.

(2003). The success case method: Find out quickly what’s working and what’s not. Oakland, CA: Berrett-Koehler.

Brinkerhoff

R. O.

(2005). The success case method: A strategic evaluation approach to increasing the value and effect of training. Advances in Developing Human Resources, 7, 86-101.

Brinkerhoff

R. O.

Dressler

D. E.

(2003, May). Using the success case impact evaluation method to enhance training value & impact. American Society for Training and Development International Conference and Exhibition 2003. Symposium conducted at the meeting of American Society for Training and Development, San Diego, CA.

Brinkerhoff

R. O.

Montesino

M. U.

(1995). Partnerships for training transfer: Lessons from a corporate study. Human Resource Development Quarterly, 6, 263-274.

Burke

L. A.

Hutchins

H. M.

(2007). Training transfer: An integrative literature review. Human Resource Development Review, 6, 263-296.

10.

Burke

L. A.

Hutchins

H. M.

(2008). A study of best practices in training transfer and proposed model of transfer. Human Resource Development Quarterly, 19, 107-128.

11.

Cheng

E. W.

D. C.

(2001). A review of transfer of training studies in the past decade. Personnel Review, 30, 102-118.

12.

Chiaburu

D. S.

Marinova

S. V.

(2005). What predicts skill transfer? An exploratory study of goal orientation, training self-efficacy and organizational supports. International Journal of Training and Development, 9, 110-123.

13.

Choi

Lee

(2011). An exploring of critical success factors affecting training transfer using the success case method. The Study of Vocational Education, 30, 303-326.

14.

Coryn

C. L.

Schröter

D. C.

Hanssen

C. E.

(2009). Adding a time-series design element to the success case method to improve methodological rigor: An application for nonprofit program evaluation. American Journal of Evaluation, 30, 80-92.

15.

Frankfort-Nachmias

Nachmias

(2002). Research methods in the social sciences. London, England: Arnold.

16.

Gaudine

A. P.

Saks

A. M.

(2004). A longitudinal quasi-experiment on the effects of posttraining transfer interventions. Human Resource Development Quarterly, 15, 57-76.

17.

Gegenfurtner

Veermans

Festner

Gruber

(2009). Motivation to transfer training: An integrative literature review. Human Resource Development Review, 8, 403-423.

18.

Getha-Taylor

Fowles

Silvia

Meritt

C. C.

(2015). Considering the effects of time on leadership development: A local government training evaluation. Public Personnel Management, 44, 295-316.

19.

Holton

E. F.

Bates

R. A.

Seyler

D. L.

Carvalho

M. B.

(1997). Toward construct validation of a transfer climate instrument. Human Resource Development Quarterly, 8, 95-113.

20.

Hycner

R. H.

(1985). Some guidelines for the phenomenological analysis of interview data. Human Studies, 8, 279-303.

21.

Kirkpatrick

(1976). Evaluation of training. In Craig

R. L.

(Ed.), The ASTD training and development handbook: A guide to human resource development. New York, NY: McGraw-Hill.

22.

Keller

J. E.

Morrison

G. R.

Ross

S. M.

(1994). Designing effective instruction. New York, NY: Macmillan College Publishing Company.

23.

Kontoghiorghes

(2001). Factors affecting training effectiveness in the context of the introduction of new technology—A US case study. International Journal of Training and Development, 5, 248-260.

24.

Lee

Choi

Park

(2013). A study on facilitating factors and barrier factors influencing on training transfer: By success case method. The Korean Journal of Human Resource Development Quarterly, 15, 58-84.

25.

Lee

Park

(2014). A multiple group analysis of the training transfer model: Exploring the differences between high and low performers in a Korean insurance company. The International Journal of Human Resource Management, 25, 2837-2857.

26.

Lee

C. D.

Kahnweiler

W. M.

(2000). The effect of a mastery learning technique on the performance of a transfer of training task. Performance Improvement Quarterly, 13, 125-139.

27.

Lim

D. H.

Johnson

S. D.

(2002). Trainee perceptions of factors that influence learning transfer. International Journal of Training and Development, 6, 36-48.

28.

Lim

D. H.

Morris

M. L.

(2006). Influence of trainee characteristics, instructional satisfaction, and organizational climate on perceived learning and training transfer. Human Resource Development Quarterly, 17, 85-115.

29.

Medina

Acosta-Pérez

Velez

Martínez

Rivera

Sardiñas

Pattatucci

(2015). Training and capacity building evaluation: Maximizing resources and results with success case method. Evaluation and Program Planning, 52, 126-132.

30.

Mikesell

J. L.

Wilson

J. A.

Lawther

(1975). Training program and evaluation model. Public Personnel Management, 4, 405-411.

31.

Noe

R. A.

(1986). Trainees’ attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review, 11, 736-749.

32.

Noe

R. A.

Schmitt

(1986). The influence of trainee attitudes on training effectiveness: Test of a model. Personnel Psychology, 39, 497-523.

33.

Olivero

Bane

K. D.

Kopelman

R. E.

(1997). Executive coaching as a transfer of training tool: Effects on productivity in a public agency. Public Personnel Management, 26, 461-469.

34.

Phillips

Stone

(2002). How to measure training results: A practical guide to tracking the six key indicators. New York, NY: McGraw Hill Professional.

35.

Pynes

(2013). Human resources management for public and nonprofit management: A strategic approach. San Francisco, CA: Jossey-Bass.

36.

Salas

Rozell

Mullen

Driskell

J. E.

(1999). The effect of team building on performance: An integration. Small Group Research, 30, 309-329.

37.

Silberman

Auerbach

(2006). Active training: A handbook of techniques, designs, case examples, and tips (3rd ed.). San Francisco, CA: Pfeiffer.

38.

Sims

R. R.

(1993). Evaluating public sector training programs. Public Personnel Management, 22, 591-616.

39.

Tannenbaum

S. I.

Yukl

(1992). Training and development in work organizations. Annual Review of Psychology, 43, 399-441.

40.

van Buuren

Edelenbos

. (2013). Organizational competence development in two public agencies in the Netherlands: The effectiveness of In-company training versus learning by doing. Public Personnel Management, 42, 385-402.

41.

Wolcott

H. F.

(1990). On seeking-and rejecting-validity in qualitative research. In Eisner

E. W.

Peshkin

(Eds.), Qualitative inquiry in education: The continuing debate (pp. 121-152).

42.

Yamnill

McLean

G. N.

(2005). Factors affecting transfer of training in Thailand. Human Resource Development Quarterly, 16, 323-344.