Abstract
The futures field has witnessed a rise in foresight training for government and business leaders. How should participants judge the quality of foresight training they receive? This article reviews the history of foresight training and explains why it is incomplete without evaluation. To conclude, practical steps are offered that foresight firms can take to upgrade their training impact through continuous improvement, based on the Kirkpatrick Training Evaluation Model.
Over the past ten years, the futures field has witnessed a rise in foresight training (FT) for government and business leaders. Where once there was only a handful of foresight graduate level and certificate programs offered by universities, today it seems numerous stand-alone training firms are offering FT certificates. Is there a way that the public might evaluate the quality of these offerings?
How should we define FT? FT is a multi-day interactive training program to equip practitioners in the use of defined foresight competencies. While the delivery of FT can be through onsite multi-day meetings, these short courses are often enhanced by blended learning via pre-readings, virtual classrooms, and fieldwork. FT in this sense is less than a graduate degree in strategic foresight and something more than a one-day workshop offered at a futures conference.
While FT empowers practitioners to use futures methods to create alternative futures, matched to dreams, plans, and solutions, the training process is not unique to futures studies. Training and development of talent, whether in public or private enterprise, is undertaken by numerous industries, including professional associations, postsecondary educational institutions, management consulting firms, and business certification programs. The 2018 Training Industry Report 1 estimates that U.S.-based institutions invest 87.6 billion in total training expenditures, with 47 billion spent on staff payroll, and 11 billion spent on outside products and services. On average, training expenditures per learner in 2018 was $986 for the year, with employees engaging in 46.7 hours of formal training per year. Government or military organizations spend the most per learner ($1,433), followed by nonprofit organizations ($1,360). Some 69 percent of training hours were delivered via blended learning in 2018, with instructor-led training dropping 7 percent since 2017 in that blended mix.
Perhaps, the most telling conclusion of the 2018 Training Industry Report is the top priorities of institutions concerning talent development. The highest priority for training in terms of allocating budget resources for 2019 is increasing the effectiveness of training programs (34%) and measuring the impact of training programs (19%).
In this context, whether one offers a one-off custom solution or a branded professional development foresight program, futurists must pay attention to training evaluation. We must ask: How should potential participants judge the quality of FT? How should facilitators judge the quality of their training on foresight competencies, whether that is a one-off client custom solution or a branded professional development program? and How does FT demonstrate a return on investment (ROI) or at least reasonable ROE, or “return on expectations,” to our clients?
This article will review the history of FT, explain why FT is incomplete without evaluation, present the Kirkpatrick Training Evaluation Model, and conclude with practical steps that foresight firms can take to elevate their training through continuous improvement.
Enter the Future Workshop
The history of FT can be traced back to the creation of “The Future Workshop” by Robert Jungk (1913–1994), a German visionary who was one of the founders of the World Futures Studies Federation. As Jim Dator (2002) explains, the field of futures studies had many pioneers, such as Ossip Flechtheim, who first invented futurology, or Fred Polak, who introduce the concept of “images of the future.” But it was Jungk in the 1950s who demonstrated how practical future-oriented action could be envisioned and prototyped through effective workshops.
In Jungk’s 1987 book with Norbert Müllert, Future Workshops: How to Create Desirable Futures, he blocks FT into five phases. In Phase 1, the Preparatory Phase, the futurist decides on the topic and makes the arrangements, well before the workshop begins. The workshop encompasses three phases: a Critique Phase, a Creative Phase, and an Implementation Phase. Jungk writes, “The workshop itself begins with the critique phase, during which all the grievances and negative experiences related to the chosen topic are brought into the open.” There follows a Creative Phase, “in which participants come up with ideas in response to problems” that capture their dreams and alternative views. The workshop concludes with an Implementation Phase, where “participants critically assess the chances of getting their projects implemented; identifying the obstacles and imaginatively seeking ways around them to draw up a plan of action” (p. 11–12). Beyond Phases 1 to 4, Jungk defined the fifth phase of Action where follow-on action plans are carried out and monitored to effect change in the organization or to impact the community, to move toward a preferable future.
Dator (2002) claims Jungk’s “Future Workshop” had a profound impact on how the Manoa futures school would shape alternative futures experiences. Vidal (2005) describes Jungk’s approach as democratic problem solving, rooted in the tradition of Participatory Action Research, a primary social method that undergirds futures studies (Bell 1996, 189, 299). Following Jungk’s pattern, the World Futures Studies Federation hosted applied futures courses for young people almost annually to learn the methods of futures studies experientially, applied to contextual issues. 2 By the mid-80s, others would build on Jungk’s approach, including the Future Search Network, 3 which brought the whole system in the room, focused on the common ground the future could bring and emphasized taking responsibility for action (Weisbord and Janoff 1995).
Today, the field of FT is varied and diverse, as indicated in Table 1. This table is in chronological order over the past twenty years. FT ranges from broad practitioner training in foresight as conducted by the University of Houston, the Institute for the Future or Kedge’s School, to focused training in futures methodology, such as scenarios, policy innovation, or systems thinking by groups such as Manchester Institute of Innovation Research, School of International Futures, or Forum for the Future. There are also FT programs that certify users in specific framing or scanning software, such as Cognitive Edge or Joel Barker workshops.
Ten Examples of Foresight Training Programs: A Selection.
Why “Design and Deliver” Is Incomplete without Evaluation?
As successful as FT appears to be, a potential attendee cannot solely determine its value by a program’s longevity on the market, its frequency of delivery, its number of alumni, its training topics, or its delivery of a certificate of completion. These metrics in themselves may matter but taken alone they cannot justify the value of FT to an attendee or their company.
I was fortunate to begin my career as an Instructional Designer. At that time, Instructional Design was emerging in corporate training departments as a distinct field from Curriculum Development which served K-12 student classrooms. As Rose (2004) explains, even into the 1990s, these two fields emphasized different aims. Instructional Design focused on training programs, systemic design, technology delivery, and “the how” of instruction, whereas Curriculum Development embraced educating the whole person, spontaneous teaching, traditional lectures, and “the what” to teach over “the how” of teaching. Over the years, these differences have largely been narrowed between these two communities.
Today’s instructional designers embrace systematic process models to diagnosis, develop, deliver, and evaluate training. Chief among these process models is ADDIE, an acronym that stands for Analysis, Design, Development, Implementation, and Evaluation. These phases represent a flexible process for building effective training and performance support systems. Popularized by Florida State University in the mid-1970s, the ADDIE framework has become the Swiss army knife of instructional designers, while the field of educational technology still debates curriculum and instruction “no single model is emerging to replace reliance on process models as the core of designing within the field” (Boling and Smith 2018, 326).
Increasing, however, ADDIE, as a training process model, is being reconceptualized within larger change management processes, as a Human Performance Technology (HPT) model, in the context of organizations and their situated environments. Both ADDIE and HPT models envision change as a linear and an iterative progression of events. HPT-ADDIE combined model (see Figure 1), however, focuses on the larger business case of the enterprise and its adaptation to its environment (Dessinger et al. 2012). Environmental adaptation is something that futurists understand!
2012 performance improvement or human performance technology model.
What can the HPT-ADDIE model teach futurists? First and foremost, it is not enough to design and deliver a training program. One must employ HPT systems from beginning to end to develop and improve that training, based on evaluation. FT must encompass front-end analysis or needs analysis, to determine whether a company has a performance problem in foresight, followed by backward analysis or evaluation, to determine how much of a performance gap in managerial foresight been narrowed (Amsteus 2011).
How to Evaluate FT?
Explicit within the HPT-ADDIE model is E for evaluation. How should directors of FT programs evaluate the effectiveness of their training? Given that the spending priority for corporate training managers in 2019 is to increase the effectiveness of those programs (34%) and measure the impact of training programs (19%), 4 how do FT managers within those companies or external FT firms do that?
For over fifty years, the go-to model for evaluating training programs is the Kirkpatrick Model, developed by Donald Kirkpatrick (1924–2014), professor emeritus at the University of Wisconsin and past-president of the Association for Talent Development (ATD). D. L. Kirkpatrick (1994) felt it was best to look at evaluation as four levels: Reaction, Learning, Behavior, and Results (see Table 2). Level 1, “Reaction,” is normally what trainers value. We collect end-of-session participant comments. Some may even end up as “testimonials” on our brochures or web sites, sharing what participants thought of the training, before leaving the event. Level 2, “Learning,” measures an increase in participant knowledge or skill related to stated program outcomes.
The Kirkpatrick Four Levels of Evaluation.
Source. J. D. Kirkpatrick and Kirkpatrick (2010, 23).
In a 2009 interview with Chief Learning Officer magazine, Kirkpatrick shared, It used to be that trainers would feel, ‘If we get a good reaction and teach people the skills and knowledge they need, that’s all we can do. We have no control over them when they go back to their job. My philosophy is, yes, you have no control over them, but you must have an influence on them because unless that training gets used on the job, it’s worthless.
5
Kirkpatrick notes it is easy to get stuck in Levels 1 and 2, or in smile sheets and knowledge checks, and never get to Levels 3 and 4, where post-training evaluation is measured, in the implementation of community action plans (Jungk and Müllert 1987), or in HPT organizational change processes (Dessinger et al. 2012). How should we think about Level 3 or 4 evaluation as a futures field? Starting points in this regard might be Grim’s (2009) Foresight Maturity Model or Rohrbeck and Gemünden’s (2011) work on foresight related to the innovation capacity of a company. Slaughter (1996) also helps us think about Level 3-4 measures beyond short-term profitability measures, by envisioning social foresight beyond individual capacity, expressed by social institutions of foresight, that span across various sectors.
The four levels of training evaluation are well known to training and development professionals. Rarely are they used to their full potential, often to our detriment. When we fail to show the ROI of our training, at Levels 3 and 4, we are surprised when our FT contracts are not renewed, due to budget cuts. Why? J. D. Kirkpatrick and Kirkpatrick (2010, 5) claim “business executives are asking for increased profits, increased customer retention, decreased costs, and reduced risks. They are not specifically asking for a reduction in the skills gap or an increase in employee engagement.” Employing Level 3 and 4 evaluation can address this before the loss of ongoing FT contracts.
Building an Evaluation Model
Since 2004, I have been a university professor of foresight to graduate students. My work has shifted the past few years to building twenty-first century degrees, from the master to the doctoral level. Integral to building that curriculum is what I call creating alignment tables where I lay out five to seven program outcomes, three to four course outcomes, and twelve learning objectives to cover four multi-week modules. This design is incomplete with an assessment or evaluation model. That model for a degree program would designate key assignments spread across a two-year program to be evaluated by a multi-level rubric, to determine whether 80 percent or more our students met the competent threshold.
This same practice of multi-column evaluation emerged from 2004 to 2014 in the U.S. federal government. The Office of Personnel Management (OPM) is tasked to ensure the quality of employee training across all government agencies. They organized their evaluation model as a cause-and-effect “Logic Model.” They developed visual evaluation templates to help their agencies gauge the effectiveness of their hosted supervisory and managerial training programs. This multi-column template in Figure 2 follows Kirkpatrick’s levels and asks program evaluators to list training resource inputs, short-term training outputs, long-term outcomes, along with the baseline data for each evaluation method. 6
Logic model template for training design and evaluation.
How do we Improve our Training Systems?
How do we sharpen our skills to evaluate our FT programs? How do we allocate the proper investments needed for systematic evaluation? There are no silver bullets, but recent developments may point some ways forward. Here are five recommendations that can be adopted by FT programs and their trainers.
FT Programs Should Evaluate Learning Based on Foresight Competencies
In 2016, the Association of Professional Futurists (APF), under the leadership of Andy Hines, released a Foresight Competency Model, that defined six core foresight practices, that professional futurists should master: Framing, Scanning, Futuring, Visioning, Designing, and Adapting (Hines et al. 2017). These foresight competencies were related to forty-five futures methods that futurists use. The Foresight Competency Model, drafted by twenty-three APF members from four continents, could serve as the basis for a common language of Kirkpatrick Level 2 evaluation, by foresight trainers. A foresight trainers’ task force could be convened from among the selected programs in Table 1. These trainers could support each other as they develop valid learning assessment plans to demonstrate Level 2 learning based on training results, using foresight competencies in pre-test/post-test surveys. At the least, it would help the public understand the breadth and depth of FT as program directors reported which competencies their training program covers.
Foresight Trainers Should Seek Out Professional Development
Foresight trainers need continuing education and recognition as evaluation professionals. A leading professional association in this area is the American Evaluation Association (AEA). 7 Their publications, events, and interest groups offer great support in designing evaluation methods and using tools effectively. If FT firms do not want to build internal evaluation expertise, they might partner with a trained evaluator who can create a robust approach to program evaluation, both formative and summative.
For continuing education as a general learning professional, many trainers turn to the ATD for help. 8 ATD is the world’s largest non-profit organization for trainers. It uses a competency model for its professional constituency, reflecting on ten areas of expertise, such as instructional design, performance improvement, training delivery, learning technologies, evaluating learning impact, change management, managing learning programs, integrated talent management, coaching, and knowledge management. Beyond membership, ATD offers a Learning and Development Degree Directory, 9 for those seeking graduate education from universities in learning design and technology.
Foresight Trainers Should Create Viable Program Assessment Plans
Directors of FT need to view their programs as instructional systems that require continuous improvement based on evaluation. Great training organizations, whether external or inside institutions, optimize the value of their training by eight process areas (Harward and Taylor 2014, 8):
Strategic alignment: design learning programs that align with business objectives;
Content development: assess, design, manage, and maintain content;
Delivery: manage instructor networks and deliver via multiple modalities;
Diagnostics: identify the causes of problems and recommend solutions;
Reporting and analysis: define business metrics, report, data, and make improvements;
Technology integration: integrate learning technologies with each other, and with enterprise systems;
Administrative services: support learners via registration, technology, and other back-office functions;
Portfolio management: manage, rationalize, and maintain large training portfolios.
Building a great training organization does not happen overnight. But evidence-based evaluation across Kirkpatrick’s four levels can be foundational to program improvement.
FT Programs Should Recommend Their Alumni for Credentialing
FT programs could enhance their alumni communities, at Kirkpatrick Level-3 behavior, by directing their alumni, who become trained practitioners, to seek formal recognition of their credentials by the APF. A certificate in futures studies can validate a person’s Level-2 learning against a body of competencies, however broad or narrow. A credential issued to a futurist denotes that a body of their peers have judged them to be competent in their Level-3 behavior, related to an occupation or profession. As a community of practice, futurists have agreed at the global level to credential its professionals, rather than certify them based on coursework (Gary and von der Gracht 2015). Credentialing is a voluntary process by which a futurist receives validation of their foresight competency by a recognized body of their peers, based on an assessment of their work experience and knowledge. APF confers membership to those who can, at the minimum, claim performance in two of seven professional standards: consulting, organizational function, postgraduate degree, certificate program, speaking, teaching, and writing. 10
FT Programs Should Weigh the Value of External Accreditation and Invest in Their Future
Most FT programs are created as cottage industries to meet an immediate need. It takes scale and revenue to achieve sustainability. Those who achieve scale should consider accreditation as a training organization. Why? Accrediting bodies at the national level incentivize training organizations to conduct continuous improvement, based on an international standard (ISO/IEC 17011). 11 In the United States, the American National Standards Institute (ANSI) issues accreditation to training organizations that issue certifications to nearly one million professionals (ANSI-ISO-IEC 17024). 12
The value of accreditation for training organizations is akin to the practice of universities, who voluntarily formalize their continuous improvement cycle over three- to six-year terms, through regional accreditation. Imagine the day, when any aspiring futurist could read an annual report of an FT agency to learn how effective they are in training, and the impact their alumni have on their organizations and communities.
Where Do Foresight Trainers Go from Here?
As we approach 2020, the role of futurists continues to evolve from sages on the stage, to guides on the side, whether that is with top management teams, middle-managers, or community leaders. A new generation of foresight professionals has created “experiential futures” workshops and games (Cuhls and Daheim 2017) to train foresight champions in forward-looking organizations. This new phase of experiential futures work cannot be sustained by singular “design & deliver” models. We must see a movement from design to evaluation in FT, as basketball’s John Wooden said, “without proper self-evaluation, failure is inevitable.”
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
