A Systematic Review of Validation Practices for the Goal Attainment Scaling Measure

Search Strategy

The following six databases were searched for relevant research articles: PubMed, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Eric, PsycINFO, and Cochrane Database of Systematic Reviews. Library databases were examined using the search criteria: (a) keyword “goal attainment scaling” combined with (b) valid* (*denotes truncation to search for variations in the word). Peer-reviewed articles, written in English, published since January 1970 that describe the use of GAS with any human sample were selected. Articles over several decades were searched to gather all available literature on validation practices with the GAS. Reference lists of articles that met full-text inclusion criteria were reviewed to determine whether any additional articles should be retrieved.

Eligibility Criteria

For inclusion in this review, articles were reviewed in a step-wise manner by two reviewers, once duplicates were removed: (a) titles and abstracts were screened, followed by (b) a review of the full-text to code and select final articles. Titles and abstracts were initially screened by the first author (S.S) and a second reviewer (S.K.M) to determine whether articles identified: use of the GAS as a measurement tool and the abstract identified measurement properties of the GAS. As valid* was already searched, if any measurement properties were mentioned, the terms validity or validation would be contained in the full text. Altogether, this process was liberal and tended toward inclusion rather than exclusion. The focus was to include all articles and examine how validity was conceptualized and examined. Furthermore, experimental studies, reviews, and commentaries were all included to examine how validity evidence for the GAS was both investigated as well as interpreted. As a final screen, the first author (S.S) reviewed the full text before selecting final articles; in this last step, all articles were coded to examine how validation evidence was investigated or described in each corresponding article. The second reviewer (S.K.M) reviewed 20% of full-text articles to verify the coding and the article selection process, and to obtain a macro-level sense of similarities of ratings between both raters. Final articles were selected if the coding process verified validity evidence was examined or reviewed with the GAS. Only studies that examined validity as it pertained to measurement properties of the GAS were included, and articles that discussed social validity, ecological validity, or treatment validity were excluded. Furthermore, articles that only mentioned the GAS as “valid” but did not describe specifics about the validation process were excluded. Once data were read and coded, only those articles that described measuring validity or providing validity information for the GAS instrument were included. Figure 1 outlines details about article selection and reasons for exclusion in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol (Moher, Liberati, Tetzlaff, Altman, & PRISMA Group, 2009).

OPEN IN VIEWER

Figure 1.

PRISMA flowchart for systematic review.

Coding

A data abstraction and coding form was developed by authors to gather validity and validation information from screened articles. A coding sheet that included two sections was used to assess the available validity evidence. Section A collected descriptive information about articles and sample characteristics, and noted which articles identified as a review by their synthesis of literature, and Section B collected information about validity and validation evidence.

Using a similar coding process as Chan et al. (2014) and Cizek, Rosenberg, and Koons (2008), Section B was organized using the following scheme and the five sources identified in the Standards (AERA et al., 2014): (a) test content, (b) relations with other variables, (c) internal structure, (d) consequences, and (e) response processes (see the Appendix for a sample of the coding sheet and definitions). As well, an “other validity” category was included to account for validity sources not described in the Standards (AERA et al., 2014), and reliability was also included, since reliability is a necessary condition for validity (Hubley & Zumbo, 2013). In particular, the type of reliability estimate, specifically internal consistency, test–retest, or inter-rater estimates were noted to make a comparison with the amount of validity evidence that was sought and reported. Furthermore, to understand validity perspectives, this review coded for the following: (a) whether articles mentioned a unitary perspective of validity and (b) whether articles stated that validity was a property of the test or a property of test scores.

Data Analysis

Validity evidence that was gathered from coded articles was evaluated using a combination of inductive and deductive content analytic approaches (Elo & Kyngäs, 2008; Hsieh & Shannon, 2005; Mayring, 2000). Gathering descriptive information invoked a deductive approach as information reported in research articles were noted. A deductive content analytic approach was used to collect information on all sources of validity evidence. For instance, when gathering construct definitions for evidence based on test content, as well as evaluating the use of theory and systematic testing of individual processes provided evidence for response processes. A deductive content analytic approach was also used when information regarding validation practices was applied to coding. Evaluating sources of validity evidence and validity perspectives included a combination of approaches; a deductive approach was used to collect information that was reported, and an inductive approach was used to interpret this information.

Inter-Rater Agreement on Coded Studies

The first author and second reviewer were in agreement for 50 of 60 data points for these 37 studies, and this represented 83.3% agreement. Any differences were discussed between the first and second rater to reach consensus before assigning codes to cases.

Reporting of Validity Evidence

The majority of articles in this review reported on or examined relations to other variables (89.2%), followed by evidence based on test content (51.4%; Table 2). Evidence based on consequences was reported in one article (2.7%), and no article reported information about internal structure or response processes.

OPEN IN VIEWER

Table 2.

Frequencies of Sources of Validity Evidence Reported.

Validity evidence	No. of articles (N)	%
Test content	19	51.4
Response processes	0	0
Internal structure	0	0
Relations to other variables	33	89.2
Consequences	1	2.7

Construct representation through test content and response processes

To gather information about evidence based on test content, we reviewed articles to see whether experts were consulted, a construct was identified, and a corresponding construct definition was provided. Content was evaluated by examining goal domains, agreement between experts, comparing goals on the GAS to the content from other reports or assessments, and expert opinion. Expert panels included practitioners, patients, students, family, team members, or individuals doing intake. Although evidence based on test content included some expert consultation, the construct measured by the GAS was not clearly identified. In several articles, there was no clear construct that the GAS was identified as measuring, and articles identified the GAS in two predominant ways (Table 3): (a) solely as a measure of goals (40.5%) or (b) as both a measure of goals and its own method or measurement technique (56.7%), and one article stated the GAS construct lacked clarity and was nonspecific. While several articles mentioned a theory (13.5%) and one article identified a specific goal theory, no articles actually used a theoretic approach.

OPEN IN VIEWER

Table 3.

Description of the GAS.

GAS description	No. of articles (N)	%
Measure of goals (includes goal achievement/attainment or goal-setting)	15	40.5
Measure of goals and its own measurement technique or approach	21	56.8
Nonspecific	1	2.7

Note. GAS = goal attainment scaling.

To examine evidence based on response processes, this review evaluated whether theory was used to guide application of the GAS, and also whether response processes were empirically tested. Among the articles that mentioned a theory, only one article (i.e., Hurn, Kneebone, & Cropley, 2006) identified a specific goal theory. Furthermore, results indicate that no articles reported information about systematic testing of response processes, such as cognitive, motivational, or behavioral types of processing. Nonetheless, many articles mentioned observations or reflections on how goals were set, such as through negotiation or consensus.

Other Evidence Related to Validity

Numerous articles (37.8%) also identified other sources of validity evidence that were outside the criteria identified, such as face validity, external validity, internal validity, and congruent validity.

Validation Practices

Based on the validity evidence that was gathered, most studies tended to gather validity evidence by focusing on relations with other variables and reliability evidence. Validity evidence was often gathered as types and studies tended to gather different types of validity. No article mentioned the unitary perspective of validity or any editions of the Standards (e.g., AERA et al., 2014), and one article identified the tripartite view (i.e., Gordon, Powell, & Rockwood, 1999) as their theoretical approach to validity. Validity was discussed as either a property of the GAS measure or the GAS scores.

Validity and Validation Evidence of the GAS

Validity evidence was evaluated by all studies in a number of ways. Most validity evidence tended to focus on relations to other variables, or reliability. The concentration of data in these areas is not uncommon and is similar to the findings by Zumbo and Chan (2014). They note that the high concentration of this evidence brings some difficulty interpreting evidence and building a sound validity argument. They also note limited guidance from orientations to theory, including validity theory. Altogether, validity evidence was not gathered in any systematic way and reflected a piecemeal approach to validation.

Articles approached validity by gathering varying types of validity evidence and then reasoning that the GAS was either “valid” or “not valid.” Discussing validity in this way, as a property of the measure or its scores, implies that validity is seen as a fixed or immutable quality. This idea emerged in the 1940s, where validity was conceptualized as a static property that had to be proven or established (Goodwin & Leech, 2003). Newton and Shaw (2013) in their analysis of ways validity is discussed or thought about, suggested validity was referred to as if it were a property of a test for several reasons, such as intentional misuse, lack of awareness or misunderstanding, and genuine divergence from the view of validity as a property of interpretations. Newton and Shaw (2013) also identified 122 discrete validity labels intended to capture an aspect of validity for measurement. From these labels and the results from this review, it is apparent that articles do not consider validity as the interpretations from scores on a measure. As discovered in this review, the GAS has varying interpretations.

Interpretations of the GAS

The GAS was frequently identified as both a measure of goals and also its own measurement technique. Among articles that described the GAS solely as a measure of goals, articles either specified goals were individualized, or simply referred to goals in a broad or general manner. The term goal was used by itself or referred to related aspects of the goal construct, such as goal achievement, goal attainment, or goal-setting. It was difficult to decipher how varying aspects of the goal construct were distinguished since these terms were used interchangeably in relation to the GAS.

The GAS Score and Its Meaning

Altogether, building a validity argument is a key aspect of strengthening score interpretations. Although reliability is a part of the validity argument and provides insight into the consistency of the GAS scores, it contributes minimally to the accuracy of the findings and is not a substitute for validity (Barry, Chaney, Piazza-Gardner, & Chavarria, 2014; Zumbo & Chan, 2014). Reliability was reported in almost all reviewed articles; however, it is not enough to justify the use of the GAS score. A fundamental feature of the validity argument and integrating validity evidence within a unitary concept of construct validity is how the construct is represented (Messick, 1995).

Almost all the studies included in this review mentioned the GAS score was measuring a change, and an applied purpose of the GAS score was to produce a change score. In most cases, change was measured with respect to student or patient progress and to compare program effectiveness, for example, “program success is measured in ‘goals achieved’” (Calsyn & Davidson, 1978, p. 306). In addition, it was not uncommon for studies to interpret a particular GAS score as an evaluation of change (i.e., improvement, no change, or deterioration). Articles discussed a number of different ways the GAS evaluates change and used the terms responsiveness, sensitivity to change, and change score to discuss or denote change over time; reporting of change was highly variable and inconsistent. In a literature review that investigated how studies of treatment effectiveness and program evaluations measure change over time, the authors found there are challenges to interpreting change scores, and difficulty comparing estimates of responsiveness (Beaton, Bombardier, Katz, & Wright, 2001). Beaton et al. (2001) also noted that the same terms were used, sometimes interchangeably, and emphasize that statistics such as responsiveness are highly contextualized. Responsiveness is a term that is widely used to denote change over time or sensitivity to change (Middel & van Sonderen, 2002; Terwee, Dekker, Wiersinga, Prummel, & Bossuyt, 2003) and refers to the ability of a measure to accurately detect change when it has occurred (Beaton et al., 2001). While change scores are also considered as an indicator of change over time (Thomas & Zumbo, 2012), they may also refer more generally to any difference score (Cronbach & Furby, 1970). Although the GAS produces a score that incorporates different time points, the nature of the change needs to be understood so interpretation of the GAS score as demonstrating change is reasonable and clear.

Thus, questions about whether the GAS is a suitable measure to evaluate change hinges on clarity about its construct definition and subsequent score meaning. Before one can determine whether the GAS is a measure of change and can effectively measure change in a goal construct, understanding the interaction between participants’ responses and how they align with the goal construct is imperative. As noted by one article included in this review, “initial exposure to goal setting may have allowed the person time to reflect, thereby possibly leading to a change in the goal areas” (Rushton & Miller, 2002, p. 776). There are a number of reasons goals may change, as well as factors that influence their achievement. Although many factors relate to the goal construct, perhaps most important is a well-identified gap between goal intentions and goal behavior, demonstrated in a meta-analysis by Webb and Sheeran (2006). This discrepancy suggests that intentions do not necessarily lead to actions, and actions need facilitation (Webb & Sheeran, 2006). Indeed, activation of a goal can dissipate once a goal has been reached or if an obstacle that cannot be overcome is encountered (Fishbach & Ferguson, 2007), and activation of multiple goals can shift over time (Austin & Vancouver, 1996). Thus, if the GAS is, as Kiresuk et al. (1994) maintain, a measure of one’s “perceived ability to change” (p. 245), how does one know whether the GAS is measuring change in the identified goal or quite simply a change in goal?

Strengthening Validity Evidence and Validation Practices

Ultimately, a score cannot be interpreted on a test if one does not know what the test is measuring (Sireci, 2012), and as shown through variations in goal construct, “what the test is measuring” is unclear. Of the 37 articles included, 10 identified themselves as a review article of GAS literature; however, none appraised interpretations of the GAS or drew connections to theory. Strengthening the validity argument for the GAS requires better testing procedures to justify the goal construct measured by the GAS and verify that the GAS does measure change. As well, consequences of score interpretation and use was considered in one article (e.g., Rockwood, 1994), and more studies need to consider the applied purpose of the GAS score. Effectively, validation requires scientific inquiry alongside a rational argument to substantiate the score interpretation and use (Messick, 1995).

The variability noted among reviewed articles highlights that the GAS has many different interpretations. There is a lack of clarity regarding how the GAS is best interpreted, what specific construct the GAS measures, and whether the GAS measures a goal construct or whether the GAS is best regarded as its own measurement technique. An advantage of the unified view is that score interpretations infer a construct that underlies their score (Sireci & Sukin, 2013), and this logic can improve how the GAS is used and discussed. Given the GAS does not have items like a conventional measure that is scored, this added complexity stresses the importance of construct definitions and theory to guide how this construct is operationalized. Importantly, researchers need to gather input from students, patients, and/or families as part of response process information and validation efforts. Evidence based on response processes will enable researchers to link theoretic information and judgments about the content of a test with consistencies in item responses, thus improving explanations of score meaning and subsequent interpretations, as well as the consequences of testing (AERA et al., 2014; Messick, 1989a).

This review noted that validity evidence was often provided without explanations that enhance interpretations of the GAS measure. Articles did not employ the unitary perspective of validity that has been encouraged by the Standards (AERA et al., 2014), and did not regard validity as an integrative judgment (Messick, 1995; Zumbo, 2009). The way in which evidence was gathered and presented suggest some crucial changes are necessary to update measurement knowledge across disciplines, for better implementation and stronger collaborative practice. Perhaps most important in outlining a sound validity argument is for researchers to begin by identifying a validity theory to guide their validation approach. Researchers may legimately choose to use one validity approach over another; however, in its absence, and as shown in this review, validity evidence does not move toward the same objective. At the core of our findings is that the construct measured by the GAS is unclear and has not been substantiated by previous validity evidence. While differences will continue to exist between researchers and disciplines in choosing one view over the other, an obvious question to ask is, “How does validity evidence enhance our understanding of the GAS?” or any measure, for that matter.