Using association rules to measure Subjective Organization after Acquired Brain Injury

1.1 Psychometric properties of Subjective Organization measures

Sternberg and Tulving (1977) identified four psychometric properties essential for measuring SO: quantification, reliability, construct validity, and empirical validity. Thus, a measure of SO must be quantifiable with a numerical scale, demonstrate reproducible scoring, measure what it claims, and correlate predictably with other measures to which it is theoretically related. The current study assessed several co-occurrence measures called Association Rules (AR; Webb, 2003), as candidate indices of SO that fulfill Sternberg and Tulving’s criteria. Although AR analyses were developed for use in retail marketing, the concept of AR is generally applicable to a variety of different types of informational investigations where the goal is to uncover relationships between seemingly unrelated information. We assert, therefore, that the concept of AR is more broadly applicable to the measurement of SO than are the clustering measures summarized by Sternberg and Tulving (1977) which makes them better suited as a clinical tool.

1.2 Association Rule analysis

The concept of Association Rules is often explained via a “market basket analysis” example used by many retailers (AR, Webb, 2003). Each person’s purchases are stored as a vector of items which are analyzed in aggregate producing association rules that derive from the frequency with which two or more items are purchased together. The analysis allows the retailer to assess which products are purchased along with others. For example, the rules bacon > eggs or ice cream > cake that derive from an AR analysis of purchases indicates that if a person purchases bacon then there is a greater than chance probability that he or she will also purchase eggs. If they buy ice cream, then they are also likely to purchase cake. These are relatively simple binary rules; more complex rules are usually produced by the same analysis and the researcher must decide which rules define the optimal organization of purchases for the retailer.

By way of extrapolation, this same technique can be used to assess cognitive processes such as SO in recall. In most multi-trial recall experiments, such as the ones that Sternberg and Tulving (1977) describe, the items recalled on any trial are analogous to a vector of customer purchases. However, it is not assumed that the recall of words must be sequential in order to demonstrate SO. The fact that the words are recalled simultaneously over learning trials is sufficient to hypothesize association among them. It is therefore reasonable to suggest that if the association rules described above can be used to index relationships among purchases, then they might also be a useful index of organization for information in memory. Because there are a number of different AR measures, the goal of this study was to assess which of these candidate measures was best suited for use as a clinical measure of mental organization.

Measures of AR are based on the concept of joint probability; they assess the likelihood that two or more items co-occur. Conventional joint probability analysis can be divided into two basic types: an independent and a dependent model. The independent joint probability model (IJP) assumes that the occurrence of one event is independent of the occurrence of others. Computing the joint probability amounts to multiplying the probability of occurrence of one item times that of another, e.g., p(A)×p(B). Three and four way probabilities are simply the multiplication of the various probabilities in sequence, e.g., p(A)×p(B)×p(C), etc. The dependent probability approach (DJP) assumes that occurrence of one event affects the occurrence of others. The computation for joint dependent probabilities is therefore much more complex. For example, recall of item A given item B is P(A|B) = p(A&B)/P(B).

Conventional AR analyses divides the data into sets of “antecedent” and “consequent” items. Antecedent items are assumed to precede the consequents. In a supervised analysis, the researcher designates the antecedent and consequent items. In an unsupervised analysis, the computer learns which items are likely the best choices for antecedents and consequents. As applied to this research, unsupervised learning of unrelated words is assumed to form a network in which recall of any item can potentially instigate recall of any other. Moreover, one person’s recall network may differ markedly from another’s, hence the name “subjective organization.” The clustering measures reviewed by Sternberg and Tulving (1977) may therefore not adequately capture the complexity or variability of the system. The joint probability measures may be the only way to adequately measure the extent to which the information is organized in memory.

Most of the measures of AR are hybrids of the dependent conditional probability computations introduced above. Balcazar & Dogbey (2013) described these procedures and Table 1 presents a summary of their computations for each measure.

The measure of Support is an index of the proportion of cases in the data that include both or all of the antecedent and consequent values under consideration. Coverage is a measure of how extensively a given item occurs in the antecedent portion of the rule. Confidence(DJP) is a conditional probability of recalling item B given that the person also recalls item A. The Confidence(IJP) is the joint probability of recalling two words simultaneously under the assumption that they are independent of one another. Leverage measures the extent to which two words are recalled together versus what would be expected if the words were recalled independently. The Lift measure is an index of the predictive value of the rule relative to using no rule at all. Any of these measures of association or combinations of them may provide a useful index of SO. Relative to those described by Sternberg and Tulving (1977), each has the advantage of defining SO without constraining the computations to analysis of adjacent items. In addition, the measures are applicable to any type of recall, not just words.

The current study’s objective was to evaluate the various measures of SO outlined above to determine their suitability for diagnosing organizational deficits commonly reported by individuals diagnosed with ABI. Some recent studies of SO with ABI survivors (Parente, et al., 2011; Wilbur, et al., 2007) suggest an indirect performance measure of SO that derives from an analysis of transfer effects in an overlapping list learning task. In each of these studies, however, SO was not measured directly but was inferred from a certain patterns of recall of the overlapping words. Our goal was to provide a direct measure of SO that conformed to all of the criteria outlined by Sternberg and Bower (1977).

2.1 Participants

Data from 23 individuals diagnosed with ABI were originally collected as part of a study reported by Twum (1994). However, much of Twum’s data were never published; the data used in this study were part of his unpublished cases. The demographic data for the participants are presented in Table 2.

All of the participants had been in coma; many of which lasted for more than three weeks. All were tested individually by a licensed psychologist and the data were collected over a period of seven years. The memory test described below occurred at the end of each test battery. The research was approved by the IRB committee at Towson University.

These data were collected using procedures originally described by Buscke (1975). The Buschke Selective Reminding Task (BSRT) scoring procedure yields several different measures of memory. Total recall is the total number of items recalled over the 12 trial study/test sequence. Short Term Storage (STR) is a measure of the person’s dependence on short term memory. It is the number of words that were recalled on a given trial without any prior or subsequent recall on the next trial. Long Term Storage (LTS) is an index of the extent to which the word has been transferred into long term memory but is not necessarily available for consistent retrieval. Long Term Retrieval (LTR) is the extent to which items that are retrieved on a given trial given have also been inconsistently retrieved on other trials. Consistent Long Term Retrieval (CLTR) is the extent to which recall occurs consistently (without failure) across the remainder of the trial sequence. Delayed Recall (DR) is the number of words that were recalled one hour after the initial learning sequence.

2.3 Procedure

Using the BSRT, the examiner read 12 unrelated words, and the participants were required to recall the word list immediately after hearing the words. The examiner then read only those words the participant had missed; after which the participant was instructed to recall the entire list (including those he or she had recalled correctly plus those that were restated). The evaluator then repeated the process of reading out loud the words the participants had missed until the participant was able to recall the entire list. This procedure continued for 12 study and test trials. Therefore, each participant provided 12 sets of data which were then analyzed using the six AR measures mentioned above. These data were then correlated with the Buschke memory scores to determine which AR indices correlated best with the measures of memory.

The Support, Coverage, Confidence, Lift and Leverage values were computed using commercially available software (Magnum Opus - Webb, 2000). The confidence measure was computed two ways: as an independent joint probability (ConfidenceIJP) and as a dependent joint probability (traditional confidence measure - ConfidenceDJP). Each participant’s word data for the 12 recall trials were imported into the Magnum Opus package, which computed the average Coverage, Support, Confidence (DJP), Lift and Leverage values and the number of significant rules. The average scores for each of these AR measures for all significant rules were then correlated with the BSRT measures of memory.

We also performed post hoc tests to illustrate the versatility and discriminative utility of the SO measure. We assessed if the average confidence measure would discriminate these ABI survivors from a group of college students. Nickerson (2013) had administered the study-test recall task using the same words to 23 college students who indicated that they had never had a brain injury. A non-parametric test of significance (Mann-Whitney U statistic) indicated that the confidence measure for college students was significantly higher than the ABI survivors’ (p < 0.05). Table 4 presents the means (bolded) and the 95% confidence intervals for the confidence measure for the ABI group versus the college students. The fact that the confidence intervals do not overlap indicates that the SO measure was able to discriminate these two types of participants.

Although there are obvious differences between the college students and ABI survivors (e.g., age, educational level) the result suggests that the confidence measures can discriminate a patient from a non-patient population.

We also analyzed the data from two ABI survivors to illustrate how the SO measure could be used to assess the potential effectiveness of a clinical intervention for improving memory using imagery training (Twum, 1994). We assessed recall of two students who reported that they had experienced an ABI with coma prior to returning to college. Each of the students was 20 years old. One was a sophomore and the other was a junior. Both of them were female. Each was a full time student with at least a 3.0 GPA. The two students learned two lists of words according to a counterbalanced Latin square. One student first learned a 12 item unrelated word list over three study-test trials without imagery instructions followed by a second list with 12 different words with imagery instructions (Twum, 1994). The second student learned the imagery list first over three study test trials followed by the unrelated word list on the second series of three study-test trials. The mean and confidence measures for the imagery versus no imagery conditions are presented in Table 5. Although statistical analysis was impossible because there were only two cases, the results show that imagery did improve recall and SO relative to the no-imagery condition. The combined result suggests that the SO measure can be used to assess the potential value of a therapeutic intervention.

4.1 Conclusions

There are three major findings in these data. First, measures of association rules can be used as indices of SO. Second, the AR measures evaluated in this study accord with the various efficiency constraints proposed by Sternberg and Tulving (1977). Third, the Confidence measures are the best candidates for an index of SO whereas the Leverage computation is the worst candidate.

Although several measures of SO were proposed during the 1970’s, all of these earlier indices involved computing the number of adjacent pairs of words that were recalled in a study-test sequence. These methods do not, however, measure non-adjacent mental associations. The AR computations described above assess associations among any items that occur together consistently during learning without having, necessarily, to be recalled adjacent to one another. Further, these measures can be applied to any type of recall or item generation including recall of visual information and other forms of verbal description.

The Confidence measures conform to the constraints of an efficient index described by Sternberg and Tulving (1977). They are quantifiable (have a computed numerical value), reliable (the same data always produces the same scores), have construct validity (are indices of co-occurrence), and they demonstrate empirical validity (correlated with measures of learning, memory, and retention). Further, although our post hoc investigation of the discriminant validity compared different populations (college students and individuals with ABI), the results suggest that the confidence measure can significantly discriminate the two groups.

4.2 Curiosities

It is unclear why the leverage measure did not correlate with any of the memory measures. This result defies a simple explanation. Although the lift measure was significantly, but inversely, related to all measures of long-term retention, it was also significantly and positively related to Short-Term Retrieval. It is therefore, possible that lift is an index of an organizational process that occurs in this memory store.

4.3 Limitations

Although the confidence (IJP) computed here produced generally higher correlations with the BSRT variables relative to the confidence (DJP) measure, we recommend the latter simply because it assumes a dependent relationship among the items at recall. The confidence (DJP) measure is, however, more difficult to compute, especially when the goal is to identify three or more word rules. It will therefore require specialized software. Another limitation concerns the stability of the rules obtained from individual participants. Usually, AR models are developed on very large data sets that include not only a training sample but also a verification sample to ensure stability of the model. Although the AR measures can conceivably be computed with as few as two study and test trials, the reliability of the measures with limited data is questionable. When there are only two trials then other SO computations are more appropriate (Senkova & Otani, 2015). The AR computations used here have not been widely implemented outside of the market basket framework nor have they ever been proposed as a clinical measurement tool. Consequently, there are no norms available for the AR measures.

4.4 Practicalities and applications

There are a number of commercially available software packages that can be used to generate AR measures. For example, packages such as SPSS Modeler, Statistica, or SAS provide well developed association rule algorithms. Each of these statistical programs has their own data entry formats and output options. We prefer the Magnum Opus program because it provides significance tests for the various rules and it also identifies the number of rules that are significant. It is therefore easy to compute the average confidence value for the significant rules as well as the number of significant rules for two or more word combinations. There are also non-commercial software applications available for computing SO values for adjacent word pairs (Senkova & Otani, 2015; Kasen & Otani, 1996).

Regardless of how the clinician would actually derive the rules, perhaps the more important issue concerns how to interpret the results. Generally, larger confidence measures and more significant rules indicate more SO. If one uses all of the AR measures, then higher support and coverage also suggest better SO. High values for lift suggest greater retrieval from short-term memory. The number of rules generated is directly related to learning.

One advantage of the AR indices is that they may be used not only to assess SO with unrelated words, but also, with any type of free recall. We have described how it could be used as a clinical tool to assess the extent to which a person subjectively organizes unrelated words. However, the analysis may also be used to assess differences in SO for unrelated visual/spatial information, the usefulness of different memory strategies such as imagery or verbal categorization, and improvement in SO resulting from other therapeutic interventions.