Abstract
Creative problem solving (CPS) requires solutions to be useful and original. Typically, its operations span problem finding, idea generation, and critical evaluation. The benefits of training CPS have been extolled in education, industry, and government with evidence showing it can enhance performance. However, although such training schemes work, less is known about the specific tools used. Knowing whether a particular tool works or not would provide practitioners with a valuable resource, leading to more effective training schemes, and a better understanding of the processes involved. A comprehensive review was undertaken examining the empirical support of tools used within CPS. Despite the surprising lack of research focusing on the use and success of specific tools, some evidence exists to support the effectiveness of a small set. Such findings present practitioners with a potential resource that could be used in a stand-alone setting or possibly be combined to create more effective training programs.
Introduction
Creative Problem Solving (CPS)
A problem exists when an individual has a goal but it is not clear how this may be achieved (Duncker, 1945). This starting point is often referred to as the initial state and the solution as the goal state. In some cases, the problem may be clear, such as obtaining a cup of coffee, and in other cases the problem may be vague or ambiguous and ill-defined, with no consensus on what would represent a good solution, requiring further exploration to aid understanding (see, for example, Dillon, 1982; Getzels, 1982). Irrespective of the problem type, however, the process of problem solving typically begins when an individual invokes any “goal-directed sequence of cognitive operations” (Anderson, 1990, p. 257). There is general agreement that for this problem solving process to be considered creative it must involve novel approaches and/or ideas leading to an original and useful solution (see Carson, 2010).
The notion of CPS was informed at an early stage by the creative process framework introduced by Osborn (1953), which stemmed from the need to explicitly define the creative process and provide a structured approach to its enhancement. Initial refinements by Osborn (1963) led him to suggest that the three key stages in the CPS process were (a) fact finding, (b) idea finding, and (c) solution finding. This was further refined by Isaksen and Treffinger (1987) into (a) understanding the problem, (b) generating ideas, and (c) planning for action. Since then, this model has been explored and modified by others in a continuous effort to refine it (e.g., Buijs, Smulders, & van der Meer, 2009; Isaksen & Treffinger, 2004; Puccio, Murdock, & Mance, 2005). Alongside this, others have developed models that attempt to describe and explain the CPS process. For instance, Hoover and Feldhusen (1994) have suggested that the ability to solve problems in a creative manner also relies on the individual’s knowledge and memory as well as their cognitive style, which includes the ability to tolerate ambiguity. More recently, Baer and Kaufman (2005) put forward the Amusement Park Theory model, which attempts to account for CPS by suggesting a series of nested domains of ability, where domain-specific creativity is encapsulated within domain-general properties such as intelligence and motivation. Others have increased the number of processing stages to reflect a more finely tuned discrimination of the problem finding processes and also include criteria for selecting potential solutions and their implementation (see, for example, Basadur, Runco, & Vega, 2000; Mumford, Medeiros, & Partlow, 2012). Nevertheless, despite the various approaches to CPS, researchers agree that such processes should be considered flexibly in the sense that they may be used in an interactive and iterative manner rather than simply seen as a linear ordered set of stages.
However, although the various names of frameworks, models, and steps contained within them may have changed and developed over time, there are a number of key features that have remained constant. This includes the fact that the first step often involves the creative problem solver applying structure to the situation. This involves clarifying the problem itself and how it might be solved (Csikszentmihalyi & Csikszentmihalyi, 1988). It begins with a systematic effort to define, construct, or focus the problem finding efforts. This helps to provide information, understanding, and direction. Typically this can be followed by ideation processes aimed at coming up with as many varied and unusual options for dealing with the problem, which in turn may be followed by a stage that involves selecting and refining the options, identifying what is “useful” and then working out how best to implement it. Within each of these phases, or stages, both divergent and convergent thinking takes place and both knowledge and motivation provide a contribution (see Runco & Chand, 1995).
It is also worth noting here that the view we take of CPS is a necessarily broad one. We regard CPS as an umbrella term that encapsulates the notion of creativity and can, but need not always include other related behaviors and outcomes such as innovation and invention. It can also be used to refer to the product of human creative behavior as well as the creative processes that are involved in producing such behavior. Such an approach is to some extent a pragmatic response to those in the field that may be separating themselves by semantic issues. Where for instance, there may be two studies examining the same aspect of CPS that produce distinct results simply because they are defining and measuring creativity in different ways. On one hand, this highlights the difficulty that exists in the field when attempting to define CPS in any precise manner. On the other hand, it also draws attention to the multi-faceted nature of CPS. It is also an issue that can be problematic when attempting to make direct comparisons across two or more studies using distinct definitions and measures. This is an issue we will return to in the “Discussion” section.
Benefits of Training CPS
Given the potential usefulness of CPS, it should come as no surprise that industry often views training schemes that work to improve CPS as important assets (see Fontenot, 1993; Thompson, 2003). Alongside this, the field of education has a long-standing relationship with training programs that have been shown to elicit significant gains in terms of creativity, problem solving, as well as improved ability to cope with the pressures of studying (see, for example, Murdock, 2003; Parnes & Noller, 1973). All of this is consistent with the views espoused by governments and employers, both of which want graduate students to think “smarter” (e.g., Pithers & Soden, 2000). Indeed, it is often noted that future local and global business success is thought to be reliant on the development of such skills (T. M. Amabile, Burnside, & Gryskiewicz, 1999; Ma, 2006; Puccio, Firestien, Coyle, & Masucci, 2006). Over time, many training schemes have been put together to facilitate CPS and, in general, there is reasonable agreement in the literature that these elicit beneficial effects in terms of improved CPS ability (e.g., DeHaan, 2009; Ma, 2006; McIntyre, Hite, & Rickard, 2003; Scott, Leritz, & Mumford, 2004a, 2004b; Wang & Horng, 2002).
In this way, training in CPS can provide an explicit framework that individuals and/or groups may use to facilitate their innate creative thinking skills. The general view is that providing structure helps (see Puccio et al., 2005) and that such structure need not act as a constraint, providing what Kokotovich (2008) referred to as “structure without stricture” (p. 67). For instance, providing explicit structure and/or instructions may help to highlight strategies that in turn can be used to help obtain additional useful information. It may also be that structure helps the individual to expand their problem space (Newell & Simon, 1972), exploring new and/or otherwise unexplored areas, leading to greater fluency, which in turn produces both a greater number of ideas and more original ideas. The structure and/or guidance often come in the form of specific “tools,” for example, brainstorming (Osborn, 1948). However, although there is a consensus regarding the beneficial effects of CPS training schemes (Ma, 2006; Mansfield, Busse, & Kreplka, 1978; Scott et al., 2004a, 2004b), there is little empirical research identifying whether the specific tools used within each of the main stages (i.e., problem finding, ideation, evaluation) have any benefit or not. This is not to say that researchers have been idle in their attempts to understand the complex processes that support and mediate CPS performance (see, for example, Mumford et al., 2012). However, despite the claims made in the plethora of books and online media, the links between a particular tool and its empirical basis are not always made clear by practitioners within the field of CPS (Adair, 2007; Hall & Wecker, 2008; Higgins, 2006; Michalko, 2006). This is also not to suggest that non-empirical evidence does not exist beyond the academic realm. Indeed, practitioners and creative professionals will have vast personal experience at their disposal; nevertheless, our focus here is not anecdote and intuition but experimental studies with measurable, tractable outcomes. 1 The paucity of these outcomes has been raised as a serious limitation and led to requests that researchers focus on this issue with particular emphasis on the effectiveness of the tools used (Ma, 2006). If the training schemes are viewed as providing toolkits to facilitate the CPS process, it seems an obvious point that a toolkit is only useful to the extent that the tools contained within it are useful and do the job for which they were designed.
Purpose of the Review
The terms technique and tool have been used interchangeably within the CPS literature, to refer to both structured training programs that incorporate a range of tools (e.g., Productive Thinking Programme, Feldhusen & Clinkenbeard, 1986; TRIZ [TRIZ is the Russian acronym for Teoriya Resheniya Izobretatelskikh Zadatch, which refers to the Theory of Inventive Problem Solving], Ilevbare, Probert, & Phaal, 2013) and specific individual tools, such as brainstorming (e.g., Hender, Rodgers, Dean, & Nunamaker, 2001). In this review, we focus on the use and benefit of specific single, stand-alone tools only. Such a tool would have a performance-enhancing aim by helping structure the thinking processes within the context of CPS.
Focusing on which tools can be used within CPS to enhance performance has several potential benefits and implications. First, simply knowing which particular tools are effective provides a useful resource for practitioners and educators in the field. 2 Second, such tools may then be combined to develop training programs that would clearly be underpinned by empirical research. Third, knowing which tools are useful should provide some theoretical insight into the processes involved within and across the stages of problem solving. This, in turn, could improve understanding of the CPS process. Furthermore, having some understanding of which tools work in a particular setting could provide the foundation for further exploration and refinement. For example, knowing that a specific tool is beneficial in one setting, or with one type of problem, could lead to further research exploring its wider impact in different settings with a variety of problems. However, it should not be assumed that tools for which there is no empirical support do not work. Rather, we would like to think of this lack of evidence as an empirical challenge to researchers to take up the gauntlet and examine the potential benefit of such tools. Likewise, where evidence is found, we should be mindful of the “file drawer” problem, where for any given published effect there may be a much greater number of unpublished studies that fail to elicit the effect (see, for example, Spellman, 2012). Last, we must remember that tools cannot readily be categorized as “working” and “not working” in the context of CPS because we do not yet have a single agreed measure of creativity beyond components such as number and quality and ideas (Mumford, Baughman, Threlfall, Supinski, & Costanza, 1996); given that the literature is not abundant, it will not be possible to make such fine-tuned distinctions between tools on these measures.
Method
This review utilized the Cochrane guidelines (Fink, 2010) to search the empirical research literature in an attempt to identify the various tools that have been used to support/enhance CPS. Such tools may be used at one or more of the CPS stages. A search was conducted for CPS techniques and tools that had been used within one or more of the CPS stages from the various research databases (e.g., EBSCO, JSTOR, PsychINFO, Web of Science, SCOPUS) and online sources (e.g., Google Scholar) using keywords related to creativity and CPS (e.g., creative*, problem solv*, CPS). A saturation point was reached when searches no longer provided new information and had identified 63 different stand-alone tools. All articles were then reviewed using the following simple content and methodological criteria: (a) published in a journal article, (b) written in English, (c) from any year, (d) utilizing a randomized control trial or experiment approach or a case study, (e) which identified independent variables, and (f) had outcome measures/findings. Articles were excluded if they simply mentioned a tool but did not experimentally manipulate, measure, operationalize, or observe it in action. Articles that merely outlined and/or described the hypothetical process of any given tool were also excluded, as were studies in which individual tools were combined within larger CPS training programs, as such combinations made it impossible to determine which element of the training process was responsible for eliciting an effect. This search reduced the number of tools to 12. 3
The following sections each outline the three CPS stages and the specific tools that have been shown to benefit performance in some way. Each of the sections includes a brief description of the particular tool and a review of the empirical evidence supporting its use.
Problem Finding Tools
Problem finding is viewed as the first part of the CPS process (Basadur, Graen, & Graen, 1982; Reiter-Palmon & Robinson, 2009). It includes the anticipation of problems, identifying problems where none are apparent, and structuring an ill-defined problem so problem solving efforts can proceed (Mumford, Reiter-Palmon, & Redmond, 1994; Runco & Nemiro, 1994). The tools that have been shown empirically to benefit performance on this initial stage of CPS are outlined in Table 1 and include brainstorming, mind mapping, restating the problem, the six good men, and the six thinking hats (STHs; Kokotovich, 2008; Kurtzberg & Reale, 1999; Mumford et al., 1994; Vernon & Hocking, 2014; Wu, Hwang, Kuo, & Huang, 2013).
Showing the Range of Tools That Have Been Shown to Enhance Problem Finding Performance, the Context the Tool Was Used in (i.e., Individually or Within a Group Setting), and the Various Outcomes (in This Instance “Other” Refers to the Ability to Clearly Articulate Complex Issues Surrounding a Problem).
Brainstorming
Brainstorming represents a way of generating ideas in an environment free from immediate evaluation and judgment (see Evans, 1996; Haman, 1996). According to Isaksen, Dorval, and Treffinger (2011), it is a tool that can be used at every stage of the CPS process. The aim within the initial stage of problem finding is to explore the problem scenario and consider it in as many new and/or unlikely ways as possible. For instance, Kurtzberg and Reale (1999) taught problem identification to a class of teenagers (13-14 years). This involved providing them with a summary of the general problem area and having them brainstorm to generate alternative problem statements. They found that after the training, the brainstorming group produced a greater number of problems that were classified as more relevant and utilized a wider range of categories compared with a no-training control group that did not utilize brainstorming.
This would suggest that simply exploring the problem in an open-ended manner can benefit problem finding skills.
Mind mapping
A mind map can be used as a way of visually organizing and formulating knowledge about a problem, identifying concepts and the relationships between them (Buzan, 1995). Such visual representations have also been referred to as thinking maps (Oxman, 2004) and cognitive maps (Eden, 2004). Using mind maps provides the individuals with an opportunity to structure their understanding of the problem, engaging and activating their knowledge base, as well as prior experiences, highlighting potential associations between seemingly disparate bits of information, as well as providing an “external memory” of their thoughts, all of which should generate multiple perspectives of the problem. Yin, Vanides, Ruiz-Primo, Ayala, and Shavelson (2005) have suggested there are five main types of structure. These are linear, circular, hub, tree, and network. The linear structure is considered the simplest and the network structure the most complex and non-hierarchical, and potentially the most useful when attempting to understand a problem.
Kokotovich (2008) trained novice designers to use mind maps to help provide them with a strategy for addressing a problem. This was intended to increase the individual’s creative search space which, in turn, would improve their understanding of the problem and its causes, as well as lead to more effective solutions. Furthermore, a tool that helps explicitly map out the potentially complex and dynamic interrelated issues of a problem was thought to provide a more flexible work space. Kokotovich (2008) found that those trained to use non-hierarchical mind maps were able to more clearly articulate the complex interrelations surrounding problem scenarios and identify more issues and themes of a higher quality. This, in turn, was positively related to more creative proposals on how to deal with the problem. More recently, Wu et al. (2013) found that having students use mind maps, alongside other tools, to construct a business plan and proposal led to these plans being rated by independent judges as more innovative than a control group who did not use mind mapping.
Such research suggests that a tool that helps the individual to develop a clearer and/or richer understanding of the nature and structure of a problem scenario may enable them to clearly articulate it, which should lead to better and more creative solutions. Nevertheless, it would be useful to know what type of map structure would best fit an ill-defined or ambiguous problem compared with a clearly stated one.
Restating the problem
It has been suggested that to first understand, or define, a problem, the individual needs to represent it, often in a variety of ways and utilizing a range of features and elements. The idea is that these representations must contain sufficient information for the individual to clearly define the problem. Research has suggested that individuals utilize past experience and knowledge as well as information contained in the cues of the initial problem to help them represent a problem in various ways when attempting to define it (Holyoak, 1984). The individual can then search through the representations in an attempt to identify the relevant elements that may be used to help clearly define the problem. Research by Mumford et al. (1994) showed that asking individuals to restate a problem in different ways prior to attempting to solve it produced better quality and more original solutions. More recently, Mumford et al. (1996) found that when individuals were presented with a problem, then offered a range of alternative restatements and asked to select the ones they thought would best help them deal with the problem, participants selected the restatements containing the highest quality information, which included procedures that could be followed and potential restrictions. Furthermore, it was the elements of procedures and restrictions that were also associated with the best outcomes. Hence, having an individual reframe the problem in different ways and, in particular, focus on the procedures that will be involved in the process, and the possible restrictions, is likely to yield the best outcome.
Six good men
Questions posed to gain a greater understanding of the problem may take a variety of forms and an obvious approach is to simply ask open-ended questions, such as those Kipling (1902) referred to as the “six good men.” These “men” are the six open-ended questions: who, what, when, where, how, and why (Kipling, 1902). The idea is that the individual approaches a problem with these questions in mind to structure and facilitate their understanding of the issues influencing and/or surrounding the problem, which in turn may improve their ability to identify more effective solutions. Such a view is consistent with research showing that the use of open questions elicits more diverse responses than close-ended questions (e.g., Reja, Manfreda, Hlebec, & Vehovar, 2003).
Vernon and Hocking (2014) showed that participants who used this tool in a problem construction task, which required them to restate a given problem, produced a greater number of ideas that were rated as more original compared with a no-intervention control group (see also Vernon & Hocking, 2016). The use of such open-ended questions can encourage the individual to stretch themselves and adopt multiple perspectives when dealing with a problem, which some have argued would support the later problem solving processes (McFadzean, 1998).
STHs
The STHs tool is a way for the individual to think about a specific problem in broader, distinct ways and include new and/or different concepts (de Bono, 2009). Each one of the STH is associated with a distinct color and emphasizes a particular style or type of thinking. For example, the white hat encourages the individual to focus on facts and information prompting the individual to ask questions about what information exists regarding the problem and how relevant information may be obtained. By addressing the questions associated with each of the colored hats, the individual is able to thoroughly explore a wider variety of issues, facts, implications, and alternatives, which should aid in the problem finding process. Furthermore, by changing hats, the individual can change viewpoints, which helps to ensure that they do not get stuck in their thinking patterns. Researchers have suggested that the technique helps to provide an explicit framework that will facilitate or scaffold creative thinking (Rizvi, Bilal, Ghaffar, & Asdaque, 2011; Schellens, Van Keer, De Wever, & Valcke, 2009) and is easy to learn (Childs, 2012).
Use of the STH in a problem construction task, involving restating a given problem, has been shown to increase the number of restatements, with such restatements being classified as more original compared with the responses from a no-intervention control group (Vernon & Hocking, 2014, 2016). This shows that helping to structure the problem finding process in this way can improve an individual’s creative output.
Ideation Tools
The second stage of CPS involves the creation of new ideas, or ideation, in an effort to solve the problem (Fogler, LeBlanc, & Rizzo, 2014). The key point here is that time is given over to explore the problem in an unrestrained manner, generating a variety of ideas in an imaginative and innovative way. The assumption is that there is a benefit to producing more ideas (see Runco, 2010). To some extent, this assumption is supported given the reported links between creative fluency and originality (see, for example, Feldhusen & Clinkenbeard, 1986; Wang & Horng, 2002). Given this, it is no surprise that the array of tools used to boost idea generation and selection has grown substantially over recent years, partly in response to attempts to enhance the creative potential of individuals (see, for example, Fogler et al., 2014; Isaksen et al., 2011), and partly in response to the need for creative ideas to fill the market with new products (Haman, 1996). Such tools can be used to help open the tap of creativity and boost the flow of ideas. The tools that have been empirically shown to benefit the ideation process are outlined below in Table 2 and include analogical thinking, assumption reversals, brainstorming/brainwriting, checklisting/heuristics and templates, the morphological matrix, and SCAMPER.
Showing the Range of Tools That Have Been Shown to Enhance Idea Generation, the Context the Tool Was Used in (i.e., Individually or Within a Group Setting), and the Various Outcomes.
Analogical thinking
Analogies may help to represent the problem in a different way, leading to new insights and greater understanding (see Gassmann & Zeschky, 2008). Or it may be that the current problem is similar to a previously solved problem and that by referring back to how the previous problem was solved could help solve the current one. For example, when trying to understand the complexity of the brain and how the various neurons in different regions communicate the analogy of a vast web of interconnecting units, such as would be seen in a telephone network, or the World Wide Web, could be used to help shed light on the issue.
According to Clement (1988), there are at least three types of analogy generation method. The first involves generating an analogy from a formal principle. Here, the idea is that a single equation and/or formal abstract principle applies to two or more different contexts. Analogies may be constructed by initially realizing that the current problem (A) is an example of a wider and/or more established principle (P). The analogous situation (B) is then retrieved and/or generated as another example of this established principle. The second approach involves transforming the problem by creating an analogous situation (B) by modifying the original problem (A). The third involves generating an analogy via an association where the individual may be “reminded” of an analogous case rather than having to transform the initial problem. Such methods echo the suggestion by Robertson (2001) that the two key phases in analogical reasoning are first to access a relevant analogue and second to adapt it to help solve a new problem.
The use of analogy is thought to play a key role in problem solving and use of analogies has been cited as the inspiration of many new products (see Fogler et al., 2014; Holyoak & Thagard, 1995). Such a tool is also useful if an impasse is reached (VanLehn & Jones, 1993), though may not always be used in the same way by good and poor problem solvers (see Ball, Ormerod, & Morley, 2004; VanLehn & Jones, 1993). The context within which the analogy is used can also influence the outcome (see Christensen & Schunn, 2007; Dunbar, 2001), with research suggesting that maintaining an open, enquiring attitude is essential when using analogies to help solve problems (Gassmann & Zeschky, 2008). However, it has been suggested that analogies can function in a variety of ways from helping with problem identification to problem solving (see Christensen & Schunn, 2009). Nevertheless, Gick and Holyoak (1983) have shown that people are not always good at identifying an analogy and may often require some hints. Although, Bearman, Ball, and Ormerod (2002) noted that novices are able to spontaneously produce analogies at a conceptual level, which can be used in problem solving.
Theoretically, the use of analogies is thought to help in the restructuring of existing knowledge elements to create novel structures (Gassmann & Zeschky, 2008; Hampton, 1998). Analogies may be drawn from knowledge bases similar to the initial problem, or from areas that are very different, what Halpern, Hansen, and Riefer (1990) referred to as the near and far domains. Dahl and Moreau (2002) have suggested that near analogies map both surface-level attributes and the relations between these attributes and as such are conceptually close to the original issue or problem. In contrast, far analogies have less opportunity to map surface-level attributes and rely more on common relations between attributes and are conceptually further away from the original issue or problem. Such “far analogies” may be what help to make mental leaps of imagination though they are thought to require a greater level of cognitive effort. Such a view is consistent with those who propose that conceptually far sources yield more creative ideas (e.g., Chan, Schunn, Cagan, Wood, & Kotovsky, 2011; Dahl & Moreau, 2002). However, others have found equivalent benefits when using both near and far analogies (Enkel & Gassmann, 2010; Tseng, Moss, Cagan, & Kotovsky, 2008). More recently, Chan and Schunn (2015) have shown that although using far analogies can benefit creative fluency, this does not necessarily translate into greater novelty. Thus, the type of analogy used may elicit different effects on creative output.
Nevertheless, research has consistently shown that use of analogical thinking can benefit CPS. For instance, research has shown that learning to use an analogy when attempting to solve one problem helped improve students’ problem solving ability on a separate problem compared with those who did not learn an analogy (Mayer, 1976, 1983; Mayer & Greeno, 1972). In fact, Mayer (1989) has shown that students who learnt to solve problems using an analogy later showed significantly enhanced problem solving skills compared with those who had not learnt to solve problems with an analogy. Dahl and Moreau (2002) also found that when pairs of individuals were assigned the task of designing a new product and given a limited time frame in which to come up with ideas, those who incorporated more analogies into their thinking produced more original ideas. In particular, they found that the use of analogical reasoning was especially beneficial in helping to generate ideas.
Although analogical reasoning may be difficult, it has been suggested that using both within- and between-domain analogies could be better than restricting the focus to only one type (Christensen & Schunn, 2009). For instance, Dahl and Moreau (2002) found that a higher proportion of far analogies led to designs classified as more original compared with near analogies. Whereas, Christensen and Schunn (2009) have suggested that instructions to utilize random between-domain cues could help to lead to more original output.
Such findings show that use of an analogy can play an important role in the idea generation phase of problem solving and that instructions to use as many different analogies as possible is likely to yield greater output, which in turn may lead to more original decisions.
Assumption reversal
Listing the assumptions relating to a problem and then reversing them can invoke a new perspective and may help overcome traditional thought patterns leading to greater potential insight and enhance the flow of ideas. Assumptions are often misplaced, outdated, and rarely questioned (Hender, Dean, Rodgers, & Nunamaker, 2002). As such, this technique can be helpful in providing a fresh view. According to Hender et al. (2002), the assumptions made about a particular problem may act as a potential constraint and as such when they are identified and reversed, this “artificial boundary” (p. 61) may be overcome. Furthermore, the process of linking the reversed assumptions back to the original problem acts as a stimulating “cue” to encourage the ideation of novel ideas. In this way, assumption reversal is a structured approach to ideation.
In general, the assumption reversal process proceeds across three stages:
List all the major assumptions about the problem.
Reverse each assumption in any way possible.
Use these reversals as primes/stimuli to help generate new ideas.
For example, think of some of the more obvious assumptions regarding higher education, which could include the following:
assessed with written exercises,
lectured to by professional,
physically attending a university, and
reading articles and text books
Then, reverse the assumptions one at a time: don’t attend a university—perhaps the information could be delivered online. Where information could be delivered by anyone, not necessarily an expert in the field and visual animations could be utilized more widely along with oral and practical assessments.
Indeed, Hender et al. (2002) found that participants using an assumption reversal technique generated more ideas than those using analogies. However, the ideas produced were judged to be less original than those generated using analogies. They attempted to account for this difference by suggesting a combination of effects of conceptual distance and cognitive load. The argument here is that the assumption reversal tool utilizes stimuli that are conceptually closer to the original problem compared with analogies, and it is this greater conceptual distance, found with analogies, that leads to the ideas generated by using analogies to be classified as more original. With regard to cognitive load, they suggest that use of analogies involves a greater cognitive load and, hence, fewer ideas generated. Such a pattern would account for their suggestion of an inverse relationship between cognitive load and creativity.
Brainstorming/brainwriting
Brainstorming is one of the most common ideation tools used and probably more is known about this tool than any other (see, for example, Diehl & Stroebe, 1987; Isaksen, 1998; Isaksen & Gaulin, 2005; Stroebe, Nijstad, & Rietzschel, 2010). It represents a reasonably unstructured free association approach whereby the individual focuses on the problem and then freely associates from it in an unconstrained manner to generate as many ideas as possible. The tool was originally developed by Osborn (1953) to be applied in a group setting as a way of encouraging people to solve problems creatively by seeking as many possible solutions in an atmosphere that is supportive and constructive rather than critical and inhibitory. Since its development, however, it has been successfully applied by individuals as well as groups (see, for example, Gallupe, Bastianutti, & Cooper, 1991; Hender et al., 2001; Isaksen, 1998; Parnes & Meadow, 1959). Nevertheless, since the early work of Taylor, Berry, and Block (1958) suggested that those working alone and using brainstorming could generate more ideas than a group using the same tool, there has been debate about the relative benefits of brainstorming alone, in nominal groups, or in interactive groups (see, for example, Diehl & Stroebe, 1987; Isaksen, 1998; Isaksen & Gaulin, 2005; Mullen, Johnson, & Salas, 1991). It is beyond the scope of this article to explore this debate in any detail and such explorations are not the focus of this work, which is primarily interested in whether the tool can benefit ideation or not.
As such, despite the ongoing contentions raised regarding potential differences between individual and group performance, previous work makes a strong case that this tool can significantly benefit ideation (Azadyekta, 2012; Gallupe et al., 1991; Gallupe et al., 1992; Hender et al., 2001; Isaksen & Gaulin, 2005; Parnes & Meadow, 1959; Stroebe et al., 2010; Torrance, 1970), especially if the founding principles initially outlined by Osborn (1953), which have since been developed and refined by others, are followed (e.g., Evans, 1996; Isaksen, 1998; Liu & Schonwetter, 2004).
These principles are as follows:
Encourage the combination and improvement of other ideas (sometimes referred to as piggybacking or hitch-hiking);
Encouraged to freely associate any ideas without boundaries;
Groups should be led by an experienced facilitator;
Information should be recorded, ideally without interrupting the flow;
Need to clearly identify the problem at the outset;
No critical assessment or evaluation of ideas; and
Quantity of ideas is the primary goal.
Alongside these general principles, it has also been suggested that sufficient time should be provided to allow the initial usual and/or stale ideas to be generated prior to eliciting the more creative ones (Haman, 1996). Also, that short breaks during the process may help to re-focus attention (Dennis, Aronson, Heninger, & Walker, 1999) and that providing some direction during the brainstorming process can not only lead to an increase in the number of solutions but also produce solutions that are classified as more creative (Santanen, Briggs, & De Vreed, 2004; for more ideas see also, Stroebe et al., 2010).
Brainwriting refers to a modified version of brainstorming that was developed to allow individual thinking space in group ideation processes (Geschka, Schaude, & Schlicksupp, 1976). According to Isaksen et al. (2011), such a tool may be useful in overcoming the dominant member effect, whereby a group may be dominated by an extroverted individual (also referred to as production blocking; Diehl & Stroebe, 1987), it can also provide privacy for individuals and, hence, may make them feel less exposed when generating more unusual ideas, which may allow them to be more comfortable. In addition, it can also provide extra time for the reflection and incubation of ideas. The procedure itself involves each person being given a sheet of paper that clearly identifies the problem and contains a grid, usually with three columns. Each person is then required to write three ideas on his or her sheet and then either pass the sheet to the person on the right or return it to a central pile (others have suggested this approach be used with drawings or sketches; see, for example, Kulkarni, Summers, Vargas-Hernandez, & Shah, 2000; Wodehouse & Ion, 2012). In this way, members of the group exchange sheets containing different ideas and when they receive or pick a new sheet, they read the previous ideas and then add three new ideas of their own before again passing it on. The exchange process continues, each person building on previously written ideas, until a specified time limit is reached or the sections of the grid are all full. When using this tool, Paulus and Yang (2000) incorporated the use of different colored pens for each person in the group so that following the ideation session, the source of ideas could be identified, increasing accountability and reducing the possibility of social loafing. Fogler et al. (2014) suggested that such a tool can be used individually as well as within groups. The process remains largely the same except that the individual simply writes down as many ideas as quickly as possible without attempting to evaluate them in any way.
Such a process is consistent with the view espoused by Janis and Mann (1977) that the sharing of ideas, which can provide a potential for cognitive stimulation, can lead to the development of creative solutions (see also Brown, Tumeo, Larey, & Paulus, 1998). Indeed, there is evidence to show that the sharing of written ideas using the brainwriting tool can lead to better ideation performance compared with verbal discussions (Heslin, 2009; Madsen & Finger, 1978; Van de Ven & Delbecq, 1974), and the sharing of visual sketched ideas also produces a greater number of ideas with better novelty value (Wodehouse & Ion, 2012). Furthermore, Paulus and Yang (2000) reported that using brainwriting to share ideas led to a greater number of unique ideas being developed compared with those who wrote down ideas but did not share them. More recently, Dugosh and Paulus (2005) extended this to show that increased exposure to ideas led to a greater number of ideas and more unique ideas generated.
Such research provides clear evidence that both brainstorming and brainwriting can improve ideation. With regard to brainstorming, the effects of this tool may also be enhanced by allowing sufficient time to explore the problem, providing short breaks as well as providing additional directions for focus. Brainwriting, however, may help to counter the possible production blocking caused by dominant group members and can be utilized in both written and visual formats.
Checklisting/forced fitting/heuristic cards/templates
The tools of checklisting, force fitting, heuristic cards, and templates all involve making or forcing connections between the problem and a selection of stimuli. These stimuli, which may be words and/or pictures can be randomly selected and often have no deliberate link to the problem. In fact, increasing the conceptual distance between the problem and the to-be-linked/associated item may help to overcome any initial blocks and break away from more traditional/orthodox ideas (Daly, Christian, Yilmaz, Seifert, & Gonzalez, 2012; Isaksen et al., 2011). The specific stimuli used can be adapted from lists generated by others that pose specific questions, such as what would happen if the problem were multiplied or made larger (e.g., idea-spurring questions; Osborn, 1963). Alternatively, it could include prompts such as “change the color” or “rearrange the parts” or simply be lists of words such as colors or shapes (see Warren & Davis, 1969). It may comprise a set of idea cards, each one containing a simple heuristic, such as “utilize opposite surface,” along with an example of the heuristic in action (see Daly et al., 2012; Wölfel & Merritt, 2013), or provide a range of templates with each one describing a situation or experience (e.g., pictorial analogy template; extreme situation template; see Goldenberg, Mazursky, & Solomon, 1999). Such tools would allow the individual to consider each item on a list, cue on a card or template as a possible source of inspiration with respect to the problem. In this way, the items/information may act as primes to help stimulate connections between unrelated points, encouraging the process of idea generation. According to Daly et al. (2012), such primes, or prompts, provide cognitive short-cuts and help create variety in ideas. Furthermore, the potential for innovative outcomes increases as more and more varied ideas are produced. Hence, they provide a specific and systematic method for generating creative ideas by helping to explore the problem space guiding the individual to generate non-obvious ideas.
According to Azadyekta (2012), using such tools to stimulate idea generation can be viewed as a game, which may provide an intrinsic level of motivation to individuals, which in turn could help them to produce more ideas of a higher quality. Indeed, early research has shown that such tools can benefit both the quantity and quality of idea generation when the participant is given only a limited amount of time (Warren & Davis, 1969). This finding would suggest that these tools could be especially beneficial if used when time is of the essence. Intriguingly, Davis and Roweton (1968) found that a shorter list of stimuli, containing only seven items, led to the production of more ideas of a higher quality than a longer list containing a combination of 54 colors, shapes, and materials. This somewhat counterintuitive finding was accounted for in terms of the shorter list providing a clearer and more focused direction regarding idea generation. Thus, it may not be the case that more stimulation in terms of longer lists is better. Following on from this, Warren and Davis (1969) found that a short list containing only seven items led to the production of more ideas rated as better quality compared with a no-intervention control group. Roweton (1969) also found that having the checklist of items available throughout the ideation stage helped to produce more ideas compared with seeing the checklist briefly at the start of the ideation process. With regard to the use of heuristic ideation cards, Daly et al. (2012) found that novice engineering students utilizing a selection of such cards produced more creative solutions when given a design task and asked to generate as many concepts as possible. Goldenberg et al. (1999) also reported that the use of templates led to the production of more creative ideas during an idea generation task compared with a no-intervention control group.
Overall, such tools can be used to help generate novel or unexpected combinations of ideas that in turn may help to create original and practical solutions. Nevertheless, Isaksen et al. (2011) added a note of caution stating that such tools can lead to the generation of highly unusual and/or challenging ideas, which may then require a “substantial refinement or development to become workable options” (p. 102).
Morphological matrix
The morphological matrix is an idea generation tool that can be used to help systematically explore a complex problem by identifying the parameters or challenges inherent within the problem in an effort to generate ideas, which can then be used and/or linked to provide possible solutions (Isaksen et al., 2011; Rickards, 1980). The tool requires that a problem initially be broken down into its various elements or dimensions. For instance, Michalko (2006) has suggested that if the problem were “how to design a new laundry hamper,” this could be broken down into the elements of product material, shape, finish, and position. There is no clear set of rules regarding how a particular problem should be broken down, or what the optimum number of elements may be. However, Michalko (2006) suggested that when identifying the elements of a problem, a good rule of thumb is to ask whether the problem would still exist “without the parameter” (p. 118) under consideration.
With regard to the structure of the matrix, again there is no clear consensus on how this should be constructed. The elements or parameters of a problem could be used as column headers with ideas relating to each separate parameter listed beneath the header (see Choon-Keong, Aris, Harun, & Kean-Wah, 2012). According to Isaksen et al. (2011), when initially generating and listing ideas in this way, there is no need to think about matching them with other elements as this can be done later. Alternatively, a two-dimensional matrix can be used where the same elements of a problem are mapped out along both the x- and the y-axes and the intersecting squares of the matrix are used to help stimulate the production of novel ideas (see Arnold, Stone, & McAdams, 2008). A modification of this two-dimensional approach can be used when the problem has more than one clear dimension and then each aspect of the problem can be mapped along one axis. For example, if the problem was to create an “optimal pen,” then possible dimensions could be the shape of the pen, the materials used to construct it, the size, and so forth. Adding more dimensions creates the option of increasing the number of possible combinations, which in turn may lead to more novel ideas.
This tool has been used effectively in business (see Rickards, 1980) and in engineering design (see Motte & Bjarnemo, 2013) as a way of generating more ideas and providing a wide range of potentially novel combinations. Indeed, research has shown that its use can significantly benefit the ideation process. For instance, Warren and Davis (1969) were among the first to show that using such a tool enabled participants to produce more ideas of a higher originality and greater practicality than those elicited by a no-intervention control group. More recently, Choon-Keong et al. (2012) reported that use of a morphological matrix by student teachers led to a significant increase in the number of ideas produced compared with brainstorming.
Nevertheless, Motte and Bjarnemo (2013) pointed out that a strength of this tool, its ability to generate a large number of possible solutions by decomposing the problem, is also a potential drawback as this can lead to what they refer to as a “combinatorial explosion” (p. 1). This has stimulated ongoing research to explore possible ways of automating the search for, and evaluation of, ideas and solutions based on the morphological matrix.
SCAMPER
This tool can be used individually or within a group setting and represents an acronym for the following set of actions that should be applied to the problem: (S) Substitute, (C) Combine, (A) Adapt, (M) Modify/Magnify/Minimize, (P) Put to other uses, (E) Eliminate, and (R) Reverse/Rearrange (Eberle, 1997). These idea-spurring instructions are thought to help stimulate the flow of new ideas. The questions serve as “triggers” for other new ideas and the instruction cues can be used in any order, sequence, or combination, and need not all be used on one problem (see Isaksen et al., 2011). The rationale for such prompts is that they help to stimulate a flexible approach that includes the search and exploration of new directions incorporating a change in perspective. Research has shown that the tool is easy to use and intuitive in nature (Motyl & Filippi, 2014).
The tool has led to the generation of more feasible ideas compared with brainstorming (Lopez-Mesa, Mulet, Vidal, & Thompson, 2011) and to the production of ideas rated as more novel when compared with a no-intervention control group (Moreno, Hernandez, Yang, & Wood, 2014). Furthermore, use of the SCAMPER tool to solve a design problem resulted in more useful solutions compared with a no-intervention control group (Chulvi, Gonzalez-Cruz, Mulet, & Aguilar-Zambrano, 2013).
Hence, the use of this reasonably intuitive and easy-to-use tool has been shown to significantly benefit the novelty and usefulness of ideas produced. Nevertheless, Isaksen et al. (2011) added a note of caution by highlighting that such a tool can be interpreted too literally if individuals use the cue instruction words that form the acronym in a fixed and/or rigid manner.
Critical Evaluation
Having clearly identified the opportunity or problem and generated a number of plausible and potentially novel solutions, the final stage in the process is to determine which solution(s) should be implemented (Mumford, Mobley, Reiter-Palmon, Uhlman, & Doares, 1991). Furthermore, this processing stage needs to address the issues of implementation, identifying possible limitations and/or pitfalls, and plan how to best deal with them. It has also been suggested that individuals should keep in mind possible contingency plans in case the proposed solution is ineffective or cannot be implemented (Reiter-Palmon & Illies, 2004).
This final stage is often represented as relying predominantly on convergent thinking (see Fogler et al., 2014; Isaksen et al., 2011; Rickards, 1993) with the idea that convergent thinking helps to make useful what divergent thinking has made novel. Unlike previous stages, the process of convergent thinking has been given less attention and, in particular, there is a surprising lack of data on the use and potential benefit of tools that could be used to help. In part, this may be because it is difficult to know a priori what “the best solution” or outcome would be given the potentially infinite variety of problems. Furthermore, unlike problem finding and ideation, which are generally assessed in terms of fluency, originality, and usefulness, and so forth, it is not clear what the most effective dependent measure(s) could be. For instance, it could be that “amount of time spent” reviewing possible ideas to distill the best approach is most important. Alternatively, key outcome measures could be the ability to compare and evaluate alternative possible solutions or that the outcome(s) closely match the opinions of experts in the field. Irrespective of the measure used, research suggests that knowledge will play a critical role in the convergent thinking process, as it is a source of ideas, and provides the criteria necessary for assessing novelty (Cropley, 2006).
There are hints in the literature that tools such as repertory grids (Stringer, 1974) can differentiate between limited options, and that the use of metrics to classify the usefulness and/or originality of an idea/product can be helpful in identifying how creative it is (see, for example, Sarkar & Chakrabarti, 2008). Such classifications can be conducted using a dichotomous approach that identifies potential “winners” and “losers” or includes a Likert-type scale that attempts to rate the usefulness and/or originality of an idea/product across a wider range of classifications from poor to highly creative. Such ratings can be carried out by an expert, a group, or by “crowd sourcing” (see, for example, Bao, Sakamoto, & Nickerson, 2011), though work by Klein and Garcia (2014) suggested that crowds may be better at identifying and eliminating poor ideas as opposed to spotting the more creative ones. Nevertheless, Chulvi, Mulet, Chakrabarti, Lopez-Mesa, and Gonzalez-Cruz (2012) recently showed that when rating the ideas that emerged from a brainstorming session of groups of designers and engineers, tasked to solve a range of design problems, there was good agreement between the ratings obtained using such metrics and those given by “experts” with at least “eight years of professional experience” (p. 249).
As such, simply rating the ideas that are produced in terms of their usefulness and originality may help to separate the wheat from the chaff.
Discussion
Findings
There are several points that are clear from the above review of tools. First, many more tools have been developed to aid in the ideation stage, with fewer developed to facilitate problem finding and virtually none for evaluation. Second, using a tool can clearly enhance CPS performance, particularly during the first two stages. Such a finding is consistent with the many reviews of training schemes, which show clear benefits in performance (see, for example, Feldhusen & Clinkenbeard, 1986; Mumford et al., 1994; Scott et al., 2004a, 2004b). However, although using such tools can certainly enhance performance, the benefit may not always be in the same way. For example, use of a particular tool could improve the number of responses, or fluency, or it could improve the quality, variety, flexibility, or originality of responses. It may also benefit any combination of these measures. This highlights the range of different methods used to assess the outcome of such tools (see, for example, Sarkar & Chakrabarti, 2008; Shah, Smith, & Vargas-Hernandez, 2003; Silvia et al., 2008). Furthermore, even when there is agreement on what is measured, there may not always be agreement on how it is measured. For instance, many would agree that originality is an essential component of any output when focusing on CPS. However, such a measure may be arrived at via a consensual assessment technique (T. A. Amabile, 1996), which incorporates the use of independent experts/judges using Likert-type rating scales (e.g., Reiter-Palmon, Mumford, & Threlfall, 1998). Or, it may be calculated using a specific algorithmic formula (e.g., Sowden, Clements, Redlich, & Lewis, 2015). Such differences in the way outcomes are measured makes it difficult, if not impossible, to make any meaningful comparisons across tools. This also raises the issue of whether it would be better to consolidate the various measures into one overall measure of “creative ability” or continue to use the available selection. Alongside this, there is the issue of whether process-based measures may provide more insight into creative behaviors than outcome-based measures (e.g., Mumford et al., 1996). Such differences serve to highlight the challenges faced by researchers and practitioners alike in understanding which aspect of CPS performance a particular tool may benefit. Given the limited amount of empirical research directly focusing on the effectiveness of the tools reviewed here, we would agree with the suggestion proposed by Shah et al. (2003) that, at present, each of the different measures currently in use may provide us with additional information and it may be useful to know how one method compares with another in terms of flexibility versus originality, and so forth. Hence, a pragmatic stance would indicate that a particular tool can be deemed useful if it improves performance on any, or all, of the various CPS measures.
Theoretical Implications
With regard to understanding the theoretical role of such tools, it is not always clear what their theoretical underpinnings might be. In this way, the generic model of CPS is often used as a descriptive framework rather than a theoretical model (see, for example, Treffinger, Isaksen, & Dorval, 1994). Nevertheless, there is widespread agreement that the CPS process can be broken down into the processing stages outlined above with the individual going through each stage, possibly in an iterative manner, and potentially repeatedly cycling between stages to obtain a desired solution (Fogler et al., 2014; Isaksen et al., 2011). With this in mind, the ability to progress through each of these stages has been modeled to include a range of factors. For instance, Hoover (1990) put forward a model of problem finding/solving that initially contained three components. The first specified domain-specific knowledge and relates directly to what is known and how this information is organized in memory. The second included the use of skills and heuristics to acquire additional knowledge/information and incorporate that into the current knowledge base. The third component included metacognitive processes used to monitor, direct, and control such processes. Later, a fourth component, which Hoover and Feldhusen (1994) called “cognitive style,” was added. This referred to the ability to tolerate ambiguity, as well as an individual’s interest and motivation levels. Although the model itself was predominantly based on the findings from studies of gifted individuals, Hoover and Feldhusen (1994) pointed out that it provides a useful “reference point for the development of enhanced performance in non-gifted individuals” (p. 215). In terms of the tools reviewed above, it is clear that they provide a direct way to influence Hoover’s (1990) second component of skills and heuristics. Each tool provides an explicit way of developing and enhancing the individual’s problem finding/solving skills by helping them to structure and explore the problem space (cf. Newell & Simon, 1972). Thus, the tools provide a way of going beyond what would normally happen and explore conceptually distant domains, adopt alternative viewpoints, and provide possible links and/or primes to stimulate ideas.
A more recent proposition is the Amusement Park Theory model (Baer & Kaufman, 2005), which attempts to illuminate domain specificity in creativity using the metaphor of an amusement park. According to this model, some creative behaviors, such as producing solutions to an advanced scientific problem, rely heavily on expertise, whereas others, such as thinking up uses for brick, do not. The authors conceptualize domain-specific creativity as encapsulated within domain-general creativity, with specificity increasing as one passes through the stages to the core. The first, “initial requirements,” involves domain-general properties such as intelligence, motivation, and opportunity; in the parlance of the metaphor, these are whatever a person needs to enter an amusement park (money, transport, and so on). The second stage involves “general thematic areas” such as the arts, humanities, and the sciences; this is selecting the type of amusement park (one with exciting rides, a zoo, and so on). The third stage is “domains,” such as the hard and social sciences within the topic of “science”; this is selecting a particular park. The final stage is “micro-domains”; this is selecting a given attraction within the park. In terms of our review, there is strong evidence that structure (i.e., the elements of a tool) is important in boosting creative performance. The tools, such as the “six hats,” where each metaphorical hat represents a way of looking at a problem, serve as “hints” or “clues” that influence thought processes. This, in Amusement Park Theory terms, might work like domain-specific processing, where previous experience becomes encoded in long-term memory and serves to steer current thinking when the experience is called upon. The relatively unstructured style of thinking associated with the absence of a tool should be more like the domain-general approach, which is a more dispositional, knowledge-free process. On this interpretation, the structure afforded by a tool such as the “six hats” works as a kind of free-floating expertise, or knowledge that can be applied to the creative situation. It may not always be advantageous, however. If this free-floating “loaned” expertise happens to be a poor fit for the current creative situation, we might expect a reduction in performance relative to a less structured, domain-general approach. In other words, there might exist creative problems whose best solutions are not to be found using, for instance, a “six hats” approach. In the language of the amusement park metaphor, the individual who might have had a better time at London Zoo has found herself at Alton Towers.
Future Directions
It should be made clear that although there is some evidence that the tools reviewed here can provide a benefit, such tools do not represent a panacea for all types of CPS. It will not be the case that all tools are equally useful and/or effective. It may be that some tools are more appropriate than others for particular tasks. Following our theoretical discussion above, it will invariably be the case that any number of factors is likely to influence the benefit and outcome of using such tools, such as, the type of problem (i.e., ambiguous or well defined; see Dillon, 1982; Getzels, 1982), its level of difficulty, the context, the type of tool, and familiarity/ease of use, along with a range of individual differences, such as intelligence (e.g., Cox, 1983), knowledge of topic area (Brugman, 1995), personality (Reiter-Palmon et al., 1998), motivation (T. A. Amabile, 1996; Sternberg & Lubart, 1999), and divergent thinking ability (Lee & Cho, 2007), all of which are likely to influence the outcome of using such tools, and which may be interpreted within a range of theoretical frameworks.
Although such issues may be perceived as limitations to the generalizability of the tools and understanding their theoretical basis, we would like to think of them as an empirical challenge for future researchers. The tools outlined above have been shown to benefit performance in one or more situation and this begs the question of whether they would also work in other situations. Alongside this, there are other issues that also need to be addressed such as the ideal length and duration of training, whether it is better to use such tools individually or combine them, how long any such beneficial effect may last, whether there are clear transfer benefits across situations, and the careful selection of outcome measures that should be used to assess such impact.
Conclusion
The objective of this review was to examine the empirical basis of the available tools used within/across the stages of CPS, with a view to identifying those tools that had some empirical support and that had been clearly shown to benefit performance. The review highlights a number of interesting conclusions. First, despite the limitations mentioned above, CPS skills can be clearly enhanced by use of particular tools. Second, although there are many tools available, only a few have any empirical support. Third, the variety of tools that have been shown to benefit CPS performance do not always do so in the same way. In part, this may be due to the various different outcome measures currently in use. Finally, practitioners in the field will find this review helpful in informing their current practice. Given the empirical support for the tools identified in this review, their adoption and use could lead to more effective outcomes. Furthermore, combining the tools outlined in this review could also lead to more effective training programs. As such, this review should be seen as providing an initial step in terms of identifying specific tools that have some empirical basis. Work can now begin on assessing tools for which there are many claims but no evidence in the academic literature (e.g., Plus-Minus-Interesting, Futuring, Attribute Listing, ALUo, Paired Comparisons, and others; see Fogler et al., 2014; Isaksen et al., 2011). It would also be interesting to find out how effective a CPS training program, which combined the tools identified here, would be. To use a horse racing metaphor, it is too early to tell which horse may win the race, but we have at least identified those fit to run.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
