The evaluation of a home-based paediatric nursing service: concept and design development using the Kirkpatrick model

Abstract

Background

A new paediatric hospital-in-the-home nursing service required evaluation.

Aims

To determine whether the education and training provided for nursing staff employed in the service was effective.

Methods

This paper presents the way in which a training evaluation model supported the design and evaluation of a training programme for registered nurses working in an out-of-hospital, home-based nursing service for paediatric patients.

Results

The Kirkpatrick model provides a framework for evaluating the effectiveness of workforce training for any industry including healthcare (Kirkpatrick, 2009).

Conclusions

That the Kirkpatrick model is an appropriate framework to evaluate a nursing training programme, but it is imperative to evaluate all levels of the model to be able to ascertain the success of the training and the impact on clinical practice.

Keywords

design development evaluating evaluation Kirkpatrick model nursing education training

Evaluating health programmes

There are many reasons why we evaluate health programmes and there are many challenges associated with effective evaluation. Evaluation can (a) determine achievement of objectives related to improved health status (Capwell et al., 2000; Torrance, 1997) as well as consumer satisfaction and consumer perspectives (Randall et al., 2017); (b) assess programme implementation (Capwell et al., 2000; Dooris, 2006; Fagen, 2011); (c) provide accountability to funding bodies, the community and others (Capwell et al., 2000); (d) increase community support for initiatives (Akerman, 2002; Capwell et al., 2000); (e) contribute to the scientific base for community public health interventions (Capwell et al., 2000); (f) inform policy decisions (Capwell et al., 2000; Dooris, 2006; Fernandes, 2011); and finally (g) evaluate quality and cost-effectiveness (Russell, 2011; Torrance, 1997).

The evaluation of new nursing services needs to be comprehensive but can lack rigour. For example, a systematic review of the impact of community-based nurse-led clinics showed that although much literature has rationalised the introduction of the clinics in relation to access, few studies have evaluated whether they actually improve patient access to care (Randall et al., 2017). It is an evaluation of the impact of the training as well as the training per se for which the paediatric hospital-in-the-home (HITH) project is striving.

Background to the project

Keeping children out of hospitals and providing effective healthcare in the home is a goal set by a number of health services, including Northern Sydney Local Health District (NSLHD) in Sydney, Australia. Potential benefits include cost effectiveness (Bagust, 2002; Hansson et al., 2011; Raisch et al., 2003), reduction of potential harm associated with hospitalisation, plus the psychological benefits of recovering at home and increased family satisfaction through reduced impact of the hospitalisation of a child on their family.

Outcomes associated with reduced bed days as a direct result of paediatric patients being cared for at home include social and emotional benefits (Hansson et al., 2011) and reduced exposure to the potentially negative effects of hospitalisation, including isolation from family and peers, emotional distress (Jennings, 1994) and nosocomial infections. Efficiency and economic drivers cannot be ignored when understanding the introduction of new models of care. Safe, economical and efficient are descriptors commonly used for HITH models (Maraqa and Rathore, 2010).

NSLHD provided HITH for adult patients through its Acute Post-Acute Care (APAC) service. The district established a paediatric HITH service through collaboration between existing paediatric acute services and APAC. An essential aspect to the provision of quality and safe care for children and families in their homes was the education and training of the APAC nurses to expand their portfolio to include paediatrics. The APAC nurses are typically general-trained registered nurses with little or no specialist paediatric nursing training or qualifications. A programme of paediatric education was developed following an educational needs analysis undertaken by the APAC nurses. Existing paediatric training resources were deemed suitable. The programme covered aspects of child growth and development, anatomy and physiology, paediatric life support, the deteriorating child, medication administration to children and child protection. The formal educational aspects of the programme were complemented by the mentorship of the paediatric HITH clinical nurse specialists during home visits to families and the option of clinical experience on a children’s ward in the district for up to 5 days.

Evaluating the paediatric nursing educational programme for the predominantly adult-care trained APAC nurses was an important aspect of the evaluation of the new paediatric HITH service. A suitable evaluation model was required for both the training and the impact of the training on service delivery. The overall research plan can be viewed as a multi-phased design with multiple sampling strategies occurring both sequentially and consecutively (Creswell and Plano-Clark, 2011). This paper provides an overview of the Kirkpatrick model (KM) and gives the background on how and why it is being used to evaluate this programme. Results from the evaluation are expected to be published elsewhere, but we thought it was important to document and share the development of the design used in our evaluation.

The Kirkpatrick model

The model was created in the 1950s and is described as ‘the worldwide standard for evaluating the effectiveness of training’ (Kirkpatrick & Kayser, 2007). It can be applied before, during and after training to both maximise and demonstrate its value to the organisation. It is comprised of four components of evaluation referred to in the model as level 1 Reaction, level 2 Learning, level 3 Behaviour and level 4 Results (Figure 1). Each level can be evaluated using qualitative or quantitative methods.

Figure 1.

Kirkpatrick levels of training evaluation applied to the evaluation of the home-based paediatric nursing service.

Considerable international success has been experienced in applying KM to health services evaluation (Dubrowski and Morin, 2011; Lee et al., 2013; Nestel et al., 2011; Weller et al., 2012), management training (Omar et al., 2009), and education programmes (Anderson, 2009; Badu, 2013; Sinclair et al., 2015). The model has been used effectively in evaluating education programmes in health-related studies (Dubrowski and Morin, 2011; Lee et al., 2013; Nestel et al., 2011; Sinclair et al., 2015) that aimed to improve knowledge and change culture.

Frequently used evaluation methods across each of the four levels of evaluation in KM are presented in Table 1. The choice of methods used has been supported by the original authors (Kirkpatrick and Kirkpatrick, 2007). The importance of a standardised and consistent framework for evaluation of education and training programmes was also highlighted in the literature (Dubrowski and Morin, 2011; Santos et al., 2012).

Table 1.

Frequently used evaluation methods across each of the four levels of evaluation in the Kirkpatrick model for health-related training.

Evaluation methods	Literature
Knowledge questionnaires	Anderson, 2009; Badu, 2013; Grzeskowiak et al., 2015; Haller et al., 2008; Kar et al., 2014; Matsuda et al., 2016; Nestel et al., 2011; Omar et al., 2009; Pickworth and Snyman, 2012; Santos et al., 2012; Sinclair et al., 2015; Tennill, 2011
Satisfaction surveys	Anderson, 2009; Grzeskowiak et al., 2015; Haller et al., 2008; Kar et al., 2014; Lee et al., 2013; Sinclair et al., 2015
Rating scales	Bylund et al., 2011; Haller et al., 2008; Matsuda et al., 2016; Nestel et al., 2011
Interviews/discussions	Nestel et al., 2011; Omar et al., 2009; Pickworth and Snyman, 2012; Tennill, 2011
Observation	Anderson, 2009; Matsuda et al., 2016; Santos et al., 2012
Forums	Patel et al., 2016
Reflective journals	Anderson, 2009; Lee et al., 2013

Dubrowski and Morin (2011) argued that a criticism of many outcome-based programme evaluation models is that they fail to demonstrate successful outcomes in terms of behavioural change. Behavioural change is one of the levels of KM to evaluate the impact a training programme has on service delivery.

A criticism of the evidence could be that both level 3 (Behaviour) and level 4 (Results) are not reported by many programme evaluations published in the literature (Grzeskowiak et al., 2015; Matsuda et al., 2016). The reasons for this are unclear, but the omission of the two higher levels from some evaluations, often described as a modified KM, is in itself a criticism of the model. This gap in evaluations was highlighted in systematic reviews of pain education programmes and online-based learning on clinical behaviour (Dubrowski and Morin, 2011; Sinclair et al., 2015). Conversely, evaluations that have implemented all four levels of KM have achieved a more comprehensive understanding of the impact of the training. In 2007, the Kirkpatrick authors published a practical guide to implementing the four levels to accompany the original model. It focuses attention on what they describe as a much-neglected level 3 (Kirkpatrick and Kirkpatrick, 2007). The Kirkpatrick authors suggest that level 4 is sometimes omitted due to perceptions about it being too difficult or too expensive. These are notions shared by others (Nestel et al., 2011; Weller et al., 2012).

Programmes that have successfully implemented the four levels of the KM have demonstrated more comprehensive evaluations and subsequently more robust conclusions and recommendations for their training programmes (Anderson, 2009; Bylund et al., 2011; Nestel et al., 2011; Omar et al., 2009). Conversely, those programmes with higher levels omitted in their evaluation or utilising existing data, such as attendance, were unable to demonstrate behaviour or reaction changes as effectively and the impact of training results were often inconclusive (Badu, 2013; Grzeskowiak et al., 2015; Haller et al., 2008; Kar et al., 2014; Matsuda et al., 2016; Yu and Hill, 2011), although higher levels of participant satisfaction were described (Matsuda et al., 2016).

Evaluations that utilised levels 1–3 and omitted level 4 were focused on the individual trainee's satisfaction and knowledge attainment, without understanding of behavioural changes and organisational impacts of the training (Pickworth and Snyman, 2012; Santos et al., 2012). The importance of levels 3 and 4 is underpinned by the assumption that Behaviour and Results are a direct result of the training. This potentially simplified view does not allow for the consideration of factors other than the training. These limitations have been highlighted in a critical analysis of KM (Bates, 2004). This does not undermine the validity of the application of the KM model, but simply shows a need for awareness of its limitations when analysing the data obtained. The limitations of KM may be associated with the absence of longer-term evaluations of training.

Using the Kirkpatrick Model in practice

Evaluation has traditionally been described in the later stages of project or operational management, and as such is often not attended to for various reasons. This notion is acknowledged by the KM authors with the outcomes or impacts of training often under-evaluated, despite identifying them to be arguably the essence of the real aim of the evaluation (Kirkpatrick & Kayser, 2016). It would follow therefore that the need to plan evaluation at the outset of a project requires the initial identification of the framework, in this case the KM.

Research questions require formulation prior to methods and methodologies being determined. KM levels provided the framework for this and are detailed in Table 2. An evaluation plan for the paediatric nursing education and support programme for general-trained APAC community nurses adopted a mixed-methods approach, combining qualitative and quantitative data with the aim to achieve a greater depth of understanding and corroboration (Cresswell and Plano-Clark, 2010).

Table 2.

Research plan using the Kirkpatrick model.

Stages of Kirkpatrick model	Research questions framed around the Kirkpatrick Model	Research methodologies for each stage	Research methods for each stage
Reaction	What are the reactions of the general trained community nurses to the training provided to expand their portfolio to include paediatrics?	Quantitative	Rating scales about the different aspects of the paediatric teaching
Learning	Have the community nurses learnt the relevant paediatric nursing knowledge, as defined by the Australian College of Children’s and Young People’s Nurses Standards of Practice, as a result of the training?	Quantitative: descriptive research	Knowledge questionnaire post training
Behaviour	Have the community nurses routinely and consistently applied their new knowledge and skills to their community nursing practice with children and families?	Qualitative: phenomenology	a. Semi-structured interviews with community nursing about their practice with children and families b. Semi-structured interviews with paediatric clinical nurse specialists about the community nurses’ practice
Results	What are the consumers’ (families’) perceptions about the paediatric nursing home care received by the community nurses? Will the P-HITH service be as safe as hospital care?	Quantitative: survey	a. Family satisfaction surveys (when community nurses are supported by paediatric clinical nurse specialists in the home and when support is no longer routine) b. Incident data to ensure maintained safe service – compare number of yellow and red incidents in P-HITH and paediatric ward over same time period

P-HITH: paediatric hospital in the home.

The Kirkpatrick levels of evaluation model, Reaction, Learning, Behaviour and Results (Kirkpatrick & Kayser, 2016), is an evaluation framework ideally suited to mixed qualitative and/or quantitative methodologies for each level. Each level of evaluation requires the implementation of appropriate methodology that is complementary to, but independent of, the other levels. The research questions have been framed around the KM and are summarised in Table 2.

Reaction-level evaluation method

The importance of level 1, Reaction, in being quick and simple to assess ‘the degree to which participants find the training favourable, engaging and relevant to their jobs’ is emphasised by Kirkpatrick. The authors suggest a blended evaluation approach is appropriate for level 1 evaluation tools, which allows one to ask questions about confidence and commitment to apply what is learned, and questions related to anticipated application and outcomes (Kirkpatrick and Kayser, 2016).

The rating scale Reaction evaluation of the training is based on a Likert scale (Likert, 1932), which typically has at least a 5-point scale allowing the respondent to indicate their level of agreement or disagreement. The obvious question when developing the tool is what would be an appropriate number of points for a Likert scale? The psychometric literature suggests that having more scale points is better, but there is a diminishing return after around 11 (Nunnally and Bernstein, 1994). Kirkpatrick provides examples of Likert scales for evaluation using 5-, 6-, 7- and 10-point scales without expressing a preference for a particular number (Kirkpatrick and Kayser, 2016). The proposed rating scale for the Reaction evaluation is a learner-centred, blended evaluation, 5-point Likert scale on a delayed basis once all aspects of the paediatric training have been completed.

Learning-level evaluation method

Kirkpatrick notes that like level 1, level 2, Learning, is familiar to most training professionals as it aims to assess ‘the degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training’ (Kirkpatrick and Kayser, 2016, p.42). A form of knowledge test is identified by Kirkpatrick as a common formative method for this stage and its application can be found in the literature (Anderson, 2009; Badu, 2013; Grzeskowiak et al., 2015; Haller et al., 2008; Kar et al., 2014; Matsuda et al., 2016; Nestel et al., 2011; Omar et al., 2009; Pickworth and Snyman, 2012; Santos et al., 2012; Sinclair et al., 2015; Tennill, 2011).

The Learning evaluation, using a descriptive nursing research method, aims to collect data about the knowledge and skills attained from the various aspects of the paediatric training programme. The tool used for this Learning evaluation is the knowledge questionnaire with 24 multiple-choice questions designed around the Standards for Paediatric Practice (Australian College of Children's and Young People's Nurses, 2016), which identifies key elements for paediatric nursing training. For example, Australian College of Children’s and Young People’s Nurses Standard 2 (which demonstrates ethical practice in nursing children and young people) has questions associated with it, including ‘True or false? Children and young people have the right to participate in decision making, and as appropriate to their capabilities, make decisions about their care’ and ‘Which of the following are recommended for safe sleep for newborns to reduce the risk of sudden unexpected death in infancy?’.

Behaviour-level evaluation method

The Behaviour level is identified by Kirkpatrick as the most important of the four levels because training alone will not generate changes in practice or improved clinical outcomes. Kirkpatrick also highlights that level 3 is the most disruptive and challenging to traditional training evaluation processes. It aims to assess the degree to which participants apply what they learned during training when they are back on the job’ (Kirkpatrick and Kayser, 2016, p.49). The challenging nature of this level is supported by its absence in several studies claiming to use the Kirkpatrick levels of the training evaluation model (Dubrowski and Morin, 2011; Grzeskowiak et al., 2015; Matsuda et al., 2016; Sinclair et al., 2015).

Level 3, Behaviour, evaluates the combined effects of the learning and experiences of the individuals; it is understanding how individuals synthesise this as reflected in their behaviour in the workplace. Kirkpatrick and Kayser suggest it is probably the most significant of the levels of evaluation (Kirkpatrick and Kayser, 2016).

In this study, it is therefore an evaluation of the training from the participant’s perspective that is sought for the paediatric nursing training and its effect on their clinical practice with children. Capturing and analysing an experience from an individual perspective lends itself to descriptive qualitative approaches to research, in particular a descriptive phenomenology approach rather than interpretative one. Phenomenology is described as a methodology of descriptive qualitative research, which is concerned with the study of experience, or a phenomenon from an individual’s perspective (Giorgi, 1997, 2012). It is interested in the activities of consciousness and the objects that present themselves to consciousness. It is the study of ‘the essence of consciousness’ (Husserl, 2012).

Behaviour in the clinical environment could be observed, or experiences relayed to a researcher. The community nursing environment is in the child’s home and the patient–nurse relationship is critical. Observing Behaviour in a child’s home was not deemed appropriate, and seen as potentially detrimental to clinical relationships. The semi-structured interview method provides the ability to describe and analyses the Behaviour of the APAC staff from their perspectives and from that of their paediatric mentoring colleagues without potentially affecting the clients. The questions for the APAC nurses and the paediatric nurse specialists mirrored each other, for example APAC nurses are asked ‘How comfortable do you feel communicating with children and families?’, and the paediatric nurse specialists ‘How comfortable do you feel the APAC nurses are communicating with children and families?’.

Results-level evaluation method

The Results level is the raison d’etre for training evaluation. Kirkpatrick and Kayser (2016) posits that the degree to which targeted outcomes occur is a result of the training and the support and accountability package. They identified leading indicators to assist initiatives to stay on track, motivate training graduates and reassure stakeholders. These are quality, cost, volume/production, efficiency, compliance, employee satisfaction, safety, customer/market response and customer/consumer satisfaction.

This research focuses the Results of the training evaluation on consumer satisfaction and patient safety through survey and correlational research. A family satisfaction survey created by the authors will be offered to all families admitted to paediatric HITH in NSLHD. The survey will incorporate different methods, including Likert scales, multiple choice and open text, with the intention of a greater understanding of the Results from the consumer’s perspective.

Conclusion

The KM evaluation has been developed over the past six decades to become one of the most popular frameworks for evaluating the impact of workplace training. The purpose of this paper was to provide an analysis of the literature to date to expose the ways in which the model has been adapted for use across the large number of evaluation studies that have employed it. On the basis of the literature reviewed, we concluded that the full model has provided the most valid and comprehensive framework for evaluating training programmes. Training and education for nurses are provided regularly and routinely to update on practice changes and innovations with the fundamental aim of improving patient care. There are limited evaluations compared to the volume of training provided. The purpose of this paper was to provide the background to the way in which the evaluation of HITH in a paediatric home-based service in Sydney, Australia, was approached and share the development of the design used in our evaluation.

Key points for policy, practice and/or research

The Kirkpatrick Model may be used as an effective framework to evaluate a nursing training programme. However, it is highly recommended that the evaluation is designed using all levels of the model to be able to ascertain the success of the training and its impact on clinical practice.

Kirkpatrick can be a suitable model for the design of a nurse training program evaluation.

The model facilitates the use of both quantitative and qualitative data collection.

The higher levels of KM provide the opportunity to evaluate the impact of nurse training on clinical practice and patient outcomes.

Studies completing all 4 levels of the Kirkpatrick Model provide the most comprehensive evaluations.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Catherine Jones is a Higher Degree Masters student at the Susan Wakil School of Nursing and Midwifery at the University of Sydney, and the Service Development Manager for the Child, Youth & Family clinical network in Northern Sydney Local Health District. She was the project manager for the establishment of a paediatric hospital in the home service in Northern Sydney. She is a Registered Children's Nurse and worked for many years in various acute and critical care areas of paediatric nursing in the England and Australia.

Jennifer Fraser is an Associate Professor and has a clinical background in paediatric and child health nursing and is appointed at the Susan Wakil School of Nursing and Midwifery at the University of Sydney. She leads research to advance paediatric and child health care through a program that supports the relationship between research, policy and practice in paediatric and child health nursing. Her research activities range from testing strategies for early intervention and prevention of child abuse and neglect to identifying and responding to child abuse injury presentations in paediatric emergency settings. Her work has contributed to ongoing implementation of nurse home visiting programs throughout Australia and more recently to developing paediatric workforce capacity in child protection in Australia and internationally.

Sue Randall is Senior Lecturer and coordinator of the Masters of Primary Health Care Nursing at the Susan Wakil School of Nursing and Midwifery at the University of Sydney. Her former clinical and current academic career focusses on nursing in community and primary health care settings. Her particular research interests are in advanced nursing roles, with an emphasis on communication. This is mirrored when the focus is on people who live with chronic disease and how their self-efficacy, ability to self-manage and be true partners in care is explored. Qualitative research or mixed methods design and translation into practice are strengths in these research areas.

References

Akerman

(2002) Health promotion evaluation: Focus on healthy cities. Revista de Saude Publica 36(5): 638–646.

Anderson E (2009). Evaluating an Organizational Protocol to Implement Effective Leadership Meetings. Nova Southeastern University. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=eric&AN=ED507993&site=ehost-live Available from EBSCOhost eric database.

Australian College of Children's and Young People's Nurses (2016) Standards of Practice for Children and Young People’s Nurses, ACCYPN, Australia. https://www.accypn.org.au/knowledge-centre/standards-of-practice/.

Badu

(2013) The implementation of Kirkpatrick's evaluation model in the learning of initial value and boundary condition problems. International Journal of Learning and Development 3(5): 74–88.

Bagust

AHA

Sartain

Maxwell

et al. (2002) Economic evaluation of an acute paediatric hospital at home clinical trial. Archives of Disease in Childhood 87(6): 489–492.

Bates

(2004) A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning 27: 341–347.

Bylund

Brown

Bialer

et al. (2011) Developing and implementing an advanced communication training program in oncology at a comprehensive cancer center. Journal of Cancer Education 26(4): 604–611.

Capwell

Butterfos

Francisco

(2000) Why evaluate? Health Promotion Practice 1(1): 15–20.

Cresswell JW and Plano-Clark VL (2010) Designing and Conducting Mixed Methods Research. CA, USA: SAGE.

10.

Dooris

(2006) Healthy settings: Challenges to generating evidence of effectiveness. Health Promotion International 21(1): 55–65.

11.

Dubrowski

Morin

M-P

(2011) Evaluating pain education programs: An integrated approach. Pain Research & Management: The Journal of the Canadian Pain Society 16(6): 407–410.

12.

Fagen

(2011) Developmental evaluation: Building innovations in complex environments. Health Promotion Practice 12(5): 645–650.

13.

Fernandes

(2011) Reflections on health policy evaluation in Brazil. Cadernos de Saude Publica 27(9): 1667–1677.

14.

Giorgi

(1997) The theory, practice, and evaluation of the phenomenological method as a qualitative research procedure. Journal of Phenomenological Psychology 28(2): 235–260.

15.

Giorgi

(2012) The descriptive phenomenological psychological method. Journal of Phenomenological Psychology 43(1): 3–12.

16.

Grzeskowiak

Thomas

et al. (2015) Enhancing education activities for health care trainees and professionals using audience response systems: A systematic review. Journal of Continuing Education in the Health Professions 35(4): 261–269.

17.

Haller

Garnerin

Morales

et al. (2008) Effect of crew resource management training in a multidisciplinary obstetrical setting. International Journal for Quality in Health Care 20(4): 254–263.

18.

Hansson

Hallstrom

Kjærgaard

et al. (2011) Hospital-based home care for children with cancer. Pediatric Blood and Cancer 57: 369–377.

19.

Husserl E (2012) Ideas: General Introduction to Pure Phenomenology (Vol. First printed in 1931). Oxon, UK: Routledge.

20.

Jennings

(1994) Learning through experience: An evaluation of ‘hospital at home'. Journal of Advanced Nursing 19(5): 905–911.

21.

Kar

Premarajan

Subitha

et al. (2014) Student-centred learning in community medicine: An experience from Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry. National Medical Journal of India 27(5): 272–276.

22.

Kirkpatrick DL and Kayser W (2016) Kirkpatrick's Four Levels of Training Evaluation. Alexandria, VA, USA: ATD Press.

23.

Kirkpatrick DL and Kirkpatrick JD (2007) Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs. San Francisco, CA, USA: Berrett-Koehler Publishers Inc.

24.

Lee

Weston

Hillier

(2013) Developing memory clinics in primary care: An evidence-based interprofessional program of continuing professional development. Journal of Continuing Education in the Health Professions 33(1): 24–32.

25.

Likert

(1932) A technique for the measurement of attitudes. Archives of Psychology 140: 5–55.

26.

Maraqa

Rathore

(2010) Pediatric outpatient parenteral antimicrobial therapy: An update. Advances in Pediatrics 57(1): 219–245.

27.

Matsuda

Negishi

Otani

et al. (2016) Evaluation of an emergency treatment training program for occupational health nurses. Sangyo Eiseigaku Zasshi 58(4): 118–129.

28.

Nestel

Regan

Vijayakumar

et al. (2011) Implementation of a multi-level evaluation strategy: A case study on a program for international medical graduates. Journal of Educational Evaluation for Health Professions 8: 13.

29.

Nunnally

Bernstein

(1994) Psychometric theory. Journal of Psychoeducational Assessment 17: 275–280.

30.

Omar

Gerein

Tarin

et al. (2009) Training evaluation: A case study of training Iranian health managers. Human Resources for Health [Electronic Resource] 7, 20.

31.

Patel

Fang

Harrison

et al. (2016) Implementation and evaluation of a “works-in-progress” session to promote scholarship in an academic hospitalist group. Journal of Hospital Medicine (Online) 11(10): 719–723.

32.

Pickworth

Snyman

(2012) Changing assessment practice through in situ faculty development. European Journal of Dental Education 16(1): e59–63.

33.

Raisch

Holdsworth

Winter

et al. (2003) Economic comparison of home-care-based versus hospital-based treatment of chemotherapy-induced febrile neutropenia in children. Value in Health 6(2): 158–166.

34.

Randall

Crawford

Currie

et al. (2017) Impact of community-based nurse-led clinics on patient outcomes, patient satisfaction, patient access and cost effectiveness: A systematic review. International Journal of Nursing Studies 73: 24–33.

35.

Russell

NCC

(2011) Evaluation and measurement for improvement in service-level quality improvement initiatives. Health Services Management Research 24(4): 182–189.

36.

Santos

Vicente

Monteiro

(2012) Temporalities in evaluation of training courses: Standards and restrictions practiced by human resources professionals. Work 41(2): 217–226.

37.

Sinclair

Kable

Levett-Jones

(2015) The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: A systematic review protocol. JBI Database of Systematic Reviews and Implementation Reports 13(1): 52–64.

38.

Tennill MM (2011) Five-Year Follow-up Evaluation of a Faculty Development Program: A Qualitative Study, PhD Dissertation, University of Missouri. ProQuest, USA, Retrieved from https://search.proquest.com/docview/1012121052.

39.

Torrance

(1997) Preferences for health outcomes and cost-utility analysis. American Journal of Managed Care 3(Suppl): S8–20.

40.

Weller

Nestel

Marshall

et al. (2012) Simulation in clinical teaching and learning. Medical Journal of Australia 196(9): 594.

41.

Hill

(2011) Implementing an institution-wide resident-as-teacher program: Successes and challenges. Journal of Graduate Medical Education 3(3): 438–439.