Observation Oriented Modeling: Preparing Students for Research in the 21 st century 1

An Exercise in Causal Modeling

Zealure Holcomb's (1997) Real Data workbook is an instance of an accessible resource for students that reports numerous studies with actual data. One example (p. 123) reports data for 135 women who visited a hospital seeking evaluation for self-identified symptoms of breast cancer, such as a lump or discharge (see Lauver & Youngran, 1995 for the original article). Among other things, the goal of the study was to identify the relationship between optimism and any delay in seeking the evaluation. The authors expected a negative linear relationship between the optimism and delay values as shown in the following variable-based model:

OPEN IN VIEWER

A negative relationship was expected partly because optimism has purportedly been shown to correlate positively with better future planning and with positive attitudes of acceptance (Lauver & Youngran, 1995, p. 203).

The optimism scores ranged from 0 to 4 and were computed from responses to the original Life Orientation Test, an 8-item self-report questionnaire. The women responded to statements such as “In uncertain times I usually expect the best” and “I'm always optimistic about my future” using a 5-point rating scale. The numerical ratings on the scale were then averaged. The delay values were also reported by the women and ranged from 1 to 2,538 days (i.e., days between initial detection of the symptom and the visit to the hospital). Over half of the delay values were less than 21 days, and several extreme cases were noted, which will be discussed below.

Students should be encouraged to write down their own thoughts and questions about the posited link between optimism and delay in visiting a doctor. The following types of questions may emerge:

“Are optimistic women necessarily better planners, and why isn't planning in the model?”

“Once an optimistic woman has identified what she thinks is a cancerous lump, would she necessarily feel a sense of panic and want to immediately see a doctor, or would she be less worried?”

“Wouldn't an optimistic woman actually be more likely to judge the lump as non-cancerous and therefore delay a visit to the doctor?”

The last question would in fact lead to an opposite expected correlation, but the more general and important point is that the students' questions will likely not fit any variable-based model and Pearson correlation analysis. Why? Because to be perfectly consistent, models like the one above only permit statements about variables, such as “Relatively high values on the optimism scale are expected to be associated with lower reported delay values,” or “Optimism scores are expected to be negatively correlated with delay values (estimated r = −.20).” Again, compare these statements carefully to the ways students will naturally think about the problem. When thinking about optimists and the time it takes them to report to the doctor the students' natural focus is on the persons in the study, which is proper for discussing causes that firstly inhere in the women themselves, not in the variables and variable labels (e.g., “optimism”) constructed by the researchers. This statement is of course in line with philosophical realism, and the capability of creating consistency between how students will think naturally about the world and how they will analyze their observations is one of the main advantages of Observation Oriented Modeling and the OOM software.

By focusing on the persons in the study, the integrated whole under consideration by the students is any given woman detecting an anomaly in her breast. The integrated model (a type of iconic model) therefore begins with a person, as shown in Fig. 1. In this case, a simple stick figure is drawn to represent an individual woman in the study. Every integrated model must begin with a judgment about the integrated whole or the system under investigation, whether it be a person, a baseball team, another type of animal, an atom, a bio-chemical process, etc. As shown in Fig. 1, a mind-body distinction is made by denoting that the woman senses the lump in her breast, presumably by using her hands and fingers. In her mind an abstraction is made on the basis of what is reported by the senses. This abstraction is denoted in the figure by the circle enclosing the word “lump” and a picture of the lump. How abstraction occurs and whether or not it includes a visual component is not spelled out by the model in its current form. The woman must, however, know that her fingers have detected an internal lump rather than, for instance, an external skin tag.

OPEN IN VIEWER

Fig. 1.

Integrated model for optimism as a cause of planning a visit to the doctor.

With the lump identified, the model next incorporates optimism, and the question is raised, “What is optimism?” On the one hand, it may be considered a quality inhering in the woman and may therefore be subject to observation by another person. On the other hand, the actual observations relevant to optimism in the study are self-report. It is more appropriate, therefore, to represent optimism as an abstraction made by the woman. Similar to “this is a lump,” the abstraction is “this (myself) is a pessimist.” The problem with this part of the model, however, is that it does not fit the optimism observations which are numbers computed by averaging values attached to a scale for judgments about the self (e.g., “I'm always optimistic about my future” rated on a scale with values from 0 to 4). With scores ranging from 0 to 4, is it presumed that optimism is a continuum? Moreover, is it presumed that the woman somehow erects a crude “optimism ruler” in her mind upon which she demarcates herself? How would such a process actually occur? By contrast, why cannot one statement, such as “Generally speaking, I consider myself to be an optimist,” be used and code it as “Yes” or “No?” Why use a 5-point scale (0 to 4) rather than a 9-point scale? Does averaging items somehow improve the scores? Students can be encouraged to ask additional questions about the questionnaire in an attempt to determine what the optimism values actually mean in the context of the nature of optimism itself. They will quickly realize that theoretical, not statistical, answers are required, and they will learn that making relevant observations about optimism requires a richer theoretical understanding of exactly what it is as a quality ostensibly inhering in persons.

Proceeding with the example, nonetheless, and presuming that optimism is an abstraction drawn from the sensory and intellectual “data” of one's self, it is logically related to judgments about the lump discovered in the breast. If the woman forms an abstraction of herself as optimistic, then she will judge the lump as cancerous; whereas if she forms an abstraction of herself as pessimistic, then she will judge the lump as not cancerous. A judgment is distinguished from an abstraction by representing the former as a regular pentagon (see Fig. 1), and again exactly how the woman makes a judgment is not spelled out by the model. Once either judgment is made, then it acts as an efficient cause (arrows in the model) for a judgment regarding whether a visit to the doctor is warranted or the lump can be ignored. It can be seen in the model, however, that the lump is judged as “could be” or “is probably not” cancerous. Are these the expected judgments? Would it be an important difference if the woman judged the lump as “most certainly cancerous” or “absolutely not” cancerous? Would such judgments be an effect of optimism or some other judgment about oneself, such as fatalism? Students may also ponder if, as suggested above and contrary to the study authors' expectations, the pessimistic woman should actually be the one who judges the lump as cancerous and who rushes to make an appointment with her doctor; after all, bad things like cancer are expected to happen. All of this is left open to debate for the students, because the point is that the integrated model in Fig. 1 is much more rich and complex than the simple variable-based model above. Asking students to construct an integrated model will require them to think about the structures and processes under investigation; in other words, to think in terms of causality rather than simply causation.

Finally, outside of the woman's mind in the model, whether or not she visited the doctor is observed. Again, to be truer to the observations in hand, the model needs to be adjusted so that the number of days between detecting the anomaly and the first visit to the doctor is the “output” from the internal features of the model. Whether the structures and processes are directly observable or not, the observations acquired must reflect some commitment to them, which is partly why this overall approach is referred to as Observation Oriented Modeling.

An Initial Exercise in OOM Data Analysis

In the variable-based model, optimism and delay were assumed to be continuous quantities, hence a Pearson's correlation was computed and reported by the study authors. The correlation was significant (r = −.18, p < .05, two-tailed). It is also important to point out that the authors transformed the delay responses due to radical skewness and a number of salient outliers. The p value of course indicates that, assuming the null hypothesis is true, and a host of other assumptions have been met (including random sampling, bivariate normality, and homoscedasticity), the probability of obtaining results as extreme or more extreme than −.18 or .18 is less than .05. The conclusion to be drawn in the standard null hypothesis significance testing procedure is therefore that the two assumed continuous variables are linearly related in the population. The observed correlation serves as the estimated magnitude of the linear association, and at −.18 is rather anemic.

In the OOM software, students must first define the units of observation for the optimism and delay values. Rather than refer to optimism and delay as variables, in OOM they are referred to as “orderings” to reflect the understanding that nature is integrated and ordered in particular ways. The integrated model is supposed to reflect this ordering, both in terms of how the optimism and delay observations are ordered or structured within themselves and how they are ordered to one another. As derived by the study authors, the optimism observations are ordered into 33 units (0, 0.125, 0.250, 0.375,… 3.875, 4.00). Recall these values were computed by averaging responses on a 0–4 scale to eight items on the Life Orientation Test questionnaire. A simpler less precise ordering should be used to make the task of building and testing a pattern, as described below, more manageable for students. Twenty units work well, with scores ordered as 0–0.20, 0.21–0.40, 0.41–0.60,… 3.81–4.0. The delay observations can similarly be ordered into 20 units as shown in Fig. 2. The task here is similar to creating a grouped frequency histogram, but the overall goal is to build a model for explaining how the optimism and delay observations are to be ordered to one another. The groupings into units here is entirely ad hoc, whereas in original research how the observations are ordered will be determined by an integrated model.

OPEN IN VIEWER

Fig. 2.

Delay and optimism observations grouped into 20 units. Each asterisk indicates two persons, and observed frequencies are reported in square brackets.

With the observations grouped into their respective units, the next goal for the students is to devise a way of ordering one set of observations to the other, and the simplest and most powerful way to introduce them to this idea is through the Pattern Analysis – Crossed Observations procedure of the OOM software. By crossing the two orderings a matrix is created, as shown in Fig. 3. The question now is, if the optimism observations represent the cause, and the delay observations represent the effect, how are the effect observations ordered to (or conformed to) the cause observations? In simpler terms, what is the expected cause-effect pattern? Here the students are working visually rather than with equations, although simple equations could be used to determine a predicted pattern for discrete or continuous values. Figure 3a shows a pattern the students might suggest. The units are matched, in order, on a one-to-one basis with units ostensibly representing high optimism matched with units representing short delays in choosing to visit the doctor. Every woman in the study is expected to be found in one of the grey squares denoted in the pattern. The students can be further pressed to explain exactly why the observations (all of the women) should fall exactly in the grey units of the pattern. They may wish to build some imprecision or “wiggle room” into the expected pattern, like that shown in Fig. 3b, or they may opt for a more radical dichotomizing as shown in Fig. 3c. As debate ensues, students should be encouraged to return to the integrated model, which will again require them to take greater responsibility in thinking clearly and deeply about optimism itself (what it is, and how it must be observed) and exactly how it affects a decision to seek evaluation for an anomaly in the breast.

OPEN IN VIEWER

Fig. 3.

Expected patterns (darkened cells) for optimism and delay orderings (a–c), and actual observations for 135 women (d).

Using the imprecise pattern in Fig. 3b, the OOM software has an option for showing the observations in the figure as well. As can be seen in Fig. 3d, only about one quarter of the observations fit the expected pattern. With such a simple and elegant visual tool the students immediately realize using only an “eye test” that the prediction is a bust. Of the 135 women, only 33 observations (24.44%) were located in the expected crossed units. Without any equations, without any p values, without any statistical assumptions, and without any attempt to conjure an imaginary population, the students are also here reminded of the importance of accuracy in science. The current cause-effect pattern is clearly inaccurate, and since the pattern is derived from the integrated model developed by the students, the integrated model is deemed inaccurate as well. This is the primary question to be answered in Observation Oriented Modeling: Is the model accurate? As an intentional representation of the causes and effects inhering in the women in the current example study, the model is largely a failure.

Beyond the simple and elegant graph, the OOM software provides summary information regarding the accuracy of the model.

OPEN IN VIEWER

Classification Results
Pairs of Observations Classified According to the Defined Pattern(s)
Classifiable Pairs of Observations:	135
Correct Classifications:	33
Percent Correct Classifications:	24.44
Randomization Results
Observed Percent Correct Classified:	24.44
Number of Randomized Trials:	1000
Minimum Random Percent Correct:	13.33
Maximum Random Percent Correct:	34.07
Values >= Observed Percent Correct:	355
Model c-value:	0.35

The Percent Correct Classification (PCC) index, as noted, is only 24.44%, and this is the primary numeric result to be considered. The PCC index is computed or derived for most all of the analyses available in the OOM software, and students should be encouraged to look for it first in the program's output. As can also be seen, a probability statistic can be requested and used in an entirely secondary role. This probability statistic is referred to as a chance value, or c-value, in OOM in order to avoid confusion with the p value commonly encountered in null hypothesis significance testing. The c-value is in most instances in OOM derived from a randomization test, which is becoming more widely recognized by quantitative experts as superior to traditional p values (see Manly, 2006; Howell, 2007, Chapter 18) despite being lauded as early as 1969 (Winch & Campbell).

The c-value for the breast cancer data is computed by randomly pairing the optimism observations with the delay observations. Specifically, the optimism values for the 135 women are randomly shuffled. The predicted pattern is then applied to the randomly paired optimism and delay observations and the PCC index computed. If this PCC index equals or exceeds the PCC index for the actual data (here, 24.44), then randomized data performed just as well as the actual data in the context of the model, which is not a desirable occurrence. This process of randomizing the data and comparing PCC indices is repeated a set number of times (1000 trials in the output above) and the results tallied. The c-value is the proportion of instances in which the PCC index from the randomized data equaled or exceeded the PCC index for the actual data. As with the traditional p value, then, a low value is desirable, although no rational person would put forth or adopt a universal convention to determine “significance” (e.g., c < .05). The c-value is entirely secondary, whereas the integrated model and PCC index are primary. The c-value in the shown output for the breast cancer data is .35, indicating that for 355 of the 1000 trials, a PCC index of 24.44 or higher could be achieved by simply randomly shuffling the data. Not only is the PCC value of 24.44 unimpressive, but it is fairly ordinary or usual as well.

Lastly, in the move from aggregates to persons, students can be asked to compare their interpretation of the PCC index to the Pearson's correlation coefficient. Recall the correlation between optimism and delay for the breast cancer data was −.18. Students can be challenged to interpret the correlation coefficient in a way that can be applied to the 135 women or to any given woman in the study. The correlation can be squared as well to convert it to a proportional statistic. Again, students can be challenged to explain how accounting for 3% of the variance in one variable by the second variable applies to the 135 women or any given woman in the study. The point of asking students to interpret the correlation in this way is of course to remind them that their interpretation must be confined to the variables; i.e., the variances and covariances of the variables. There is simply no clear way to cross the bridge from the aggregate statistic (the Pearson's r) to the scores, or observations, themselves in a way that conveys the meaningfulness of the result for these women. By contrast, the PCC index is tied directly to the observations and how they are ordered, both in terms of their units and in terms of the expected pattern. Whether or not any given woman's observations are consistent with the integrated model and pattern analysis can be determined. Indeed, using options in the OOM software students can explicitly identify the 17 women whose observations fit the expected pattern. The readily interpretable PCC index can also be compared to the correlation value in this example to remind students that “statistical significance” does not indicate practical, clinical, or theoretical significance (Thompson, 2002), nor does statistical significance indicate that even a simple majority of observations fit an hypothesized model. Lastly, students can contrast this rich exercise of thinking about what the scores on the Life Orientation Test really mean and how they are causally related to observations of delay, with the relatively sterile exercise of checking statistical assumptions and interpreting the magnitude of the observed correlation coefficient according to some arbitrary convention (e.g., “according to Cohen's conventions, a correlation of −.18 represents a small effect size”).

An Exercise on Effect Sizes in the OOM Software

In his chapter on t tests, David C. Howell (2007) reports an example data set of weight gain (or loss) for anorexic girls who received therapy for their eating disorder and those who did not receive therapy. The outcome variable is the amount of weight gain, in pounds, from the beginning to the end of the treatment or therapy. In the parlance of modern research design, the grouping variable (Therapy vs Control) is the independent variable and cause, while weight gain is the dependent variable and effect. Using a standard variable-based model, the relationship between the two variables can be shown as follows:

OPEN IN VIEWER

Howell's original data were altered here so that the computed effect size would be medium according to popular conventions and in the expected direction (i.e., the mean for the Therapy group will be greater than the mean for the Control group); yet, when examined at the level of the individuals, most anorexic women would actually keep their existing weight or lose weight, contrary to prediction. The altered data are reported in Table 2, and the exercise to be performed by students is to analyze the data using the traditional null hypothesis significance testing approach and the OOM software.

OPEN IN VIEWER

TABLE 2

Weight Gain Data, Example Data, and Randomized Data For Control and Anorexia Therapy Groups

Control Group				Therapy Group
Weight Gain	Clear One	Clear Two	Random	Weight Gain	Clear One	Clear Two	Random
−7.5	−7.5	−7.5	3.8	−3.2	3.2	−7.0	−4.1
−7.0	−7.0	−7.0	−7.0	−3.1	3.1	−7.4	−4.5
−5.7	−5.7	−5.7	0.0	−2.5	2.5	4.0	0.0
−5.1	−5.1	−5.1	6.2	−2.4	2.4	2.4	6.4
−4.5	−4.5	−4.5	7.0	−2.2	6.2	−6.2	−5.1
−4.1	−4.1	−4.1	−0.2	−2.0	5.0	5.0	3.2
−3.8	−3.8	−3.8	3.0	−2.0	7.0	−7.0	−5.7
−3.8	−3.8	−3.8	−0.3	−2.0	4.0	4.0	−7.5
−3.7	−3.7	−3.7	5.5	−1.9	5.0	5.0	2.2
−3.4	−3.4	−3.4	−3.8	−1.9	2.0	2.0	3.4
−3.1	−3.1	−3.1	2.4	−1.9	2.0	2.0	5.5
−3.0	−3.0	−3.0	5.9	−1.8	3.8	3.8	2.0
−2.5	−2.5	−2.5	4.1	−1.4	2.2	2.2	−3.0
−2.2	−2.2	−2.2	4.8	−1.3	2.2	2.2	2.0
−1.8	−1.8	−1.8	6.1	−0.2	3.0	3.0	4.0
0.3	−0.3	−0.3	−3.7	−0.1	4.1	4.1	2.0
0.8	−0.3	−0.3	−2.2	1.7	2.0	2.0	−2.5
0.2	−0.2	−0.2	0.0	2.3	3.4	3.4	0.0
0.2	0.0	0.0	0.0	4.1	4.1	4.1	−1.8
0.5	0.0	0.0	6.1	4.1	6.1	−6.1	7.2
1.0	0.0	0.0	7.5	4.2	4.2	4.2	−0.3
1.0	0.0	0.0	4.2	4.8	4.8	4.8	−3.1
1.1	0.0	0.0	3.1	5.1	5.1	5.1	−3.4
1.3	0.0	0.0	4.1	5.5	5.5	5.5	−0.8
2.0	0.0	0.0	2.2	5.5	5.5	5.5	0.0
2.1	0.0	0.0	−0.5	5.9	5.9	5.9	0.0
3.5	0.0	−6.0	5.0	6.1	6.1	−6.1	0.0
3.6	−0.5	−0.5	−1.0	6.4	6.4	−6.4	0.0
3.9	−0.8	−0.8	2.5	7.5	7.5	−7.5	5.0
2.9	−1.0	−1.0	−3.8	7.2	7.2	−7.2	5.1

Weight is a continuous quantity, and the observations are independent both between and within groups. An independent samples t test is therefore suitable for these data, and the hypotheses can be written in three-valued logic form (Harris, 1997),

H 0 : μ therapy = μ control H A 1 : μ therapy > μ control < - predicted H A 2 : μ therapy < μ control .

The μ's of course represent population weights in pounds. Students will find that the Therapy group gained an average of 1.35 pounds (SD = 3.72) compared to the Control group who lost an average of 1.23 pounds (SD = 3.25), thus supporting the predicted difference (95%CI = 0.77 ≤ μ_T – μ_C ≤ 4.38) and efficacy of the therapy. The t test is statistically significant (t₅₈ = 2.86, p = .006), at the .05 level, and the magnitude of the effect (d = 0.74, η² = 0.12) is nearly large according to Cohen's widely used conventions (d = 0.80 is a large effect). Examination of the data also reveals no salient outliers, but the distributions appear to be somewhat bimodal, particularly for the therapy group, when examined with histograms. Boxplots, however, will show only moderate skewness.

Students should be required to list and evaluate or discuss the various assumptions of the independent samples t test: (1) random sampling or random assignment to groups, (2) continuous dependent variable, (3) null hypothesis is true, (4) p ≤ .05 is a reasonable cut-point for statistical significance, (5) normal population distributions, (6) observations are independent both within and between groups, and (7) homogeneity of population variances. Many of these assumptions of course underlie the validity of the observed p value, a probability the students should be expected to understand completely. An excellent resource for checking students' understanding of the p value is Oakes' (1986; see also, Gigerenzer, 2004) 6-item quiz. In the context of discussing the p value students should also be challenged to define the populations under consideration. Are they all women? Are they women from certain ethnic groups or regions? Are they women of certain ages? Are they women now living, or do the populations include women from the past and future? The point of this challenge is to remind students that psychologists are almost always working with arbitrarily defined, imaginary populations. The population means are therefore almost always non-empirical; i.e., they are values that cannot be obtained in actuality. The essential lesson to be learned from discussing assumptions and populations is that null hypothesis significance testing is an extremely abstract, mathematical-statistical routine in which the primary goal is to pursue (estimate) population parameters that rarely have any basis in lived reality.

Turning to effect size, by declaring the t test “statistically significant” and rejecting the null hypothesis, the conclusion is that the population means differ by some magnitude. Cohen's d represents a standardized estimate of that magnitude based on the sample means and pooled standard deviation. For the current data the value is computed as,

d = X ‾ therapy - X ‾ control S pooled 2 = 1.35 - ( - 1.23 ) 12.17 = 0.74.

The formula incorporates aggregate statistics and consequently has no direct bearing on any one of the women in the study or on any pair of women from the two groups. As has been made abundantly clear above, however, causes and their effects inhere in the things or entities under investigation – in this instance, the women in the two groups. Students should be challenged to relate the value of d to the women in the study. Another popular index of effect size is η², which the students should compute and again attempt to relate to the women in the study: η² = t²/(t² + df) = 2.86²/ (2.86² + 58) = 0.12. This task will prove even more difficult as the interpretation of η² is one based on shared overlap between variables. The value of 0.12 indicates that the independent and dependent variables share 12% of their variance. Conveying what this effect size index means for the actual effectiveness of the therapy for the individual women in the study is impossible, which is why psychologists eschew the problem by using Cohen's arbitrary conventions to describe their effects as small, medium, or large.

By contrast, analyzing the same data in the OOM software will remind students that causes inhere in the persons in the study, and that any discussion of effects must be relevant to these persons. The analysis begins by first defining the units of observation. The women are observed to be in one of two groups: the Therapy group or the Control group. These two units of observation represent the cause, or they at least carry information about the causal forces effecting weight gain or loss (in Observation Oriented Modeling, an integrated model must eventually be worked out to truly understand the causes). The changes in weight are continuous quantities representing a change in this quality (heaviness) over time. The values in Table 2 range from −7.5 to 7.5 pounds for the two groups of women, and most of the women have unique values. In OOM, continuous quantities must often be grouped so that each unit (value) is represented by more than one observation. Because a population parameter is not being estimated, the notion of statistical power is meaningless in OOM. Observations can consequently be grouped without fear of losing power. While modern methodologists, for instance, advise strongly against dichotomizing data (MacCallum, Zhang, Preacher, & Rucker, 2002; Irwin & McClelland, 2003), such alterations are welcome in OOM if they advance the goal of identifying clear and meaningful patterns in the observations. The weights for the two groups of women were thus organized into 8 units (–8.0 to −6.1, −6.0 to −4.1,… 4.0 to 5.9, 6.0 to 8.1). Students can be encouraged to try different groupings after the initial analysis and explore how they affect the results, as will be shown below.

The Build/Test Model option in the OOM software is then used to attempt to bring the 8-unit weight ordered observations into conformity with the 2-unit group ordered observations using binary Procrustes rotation, a procedure that does not rely on means or covariances but is instead based on the observations themselves (see Grice, 2011, Chapter 3). The analysis is ad hoc compared to the a priori pattern defining methods used with the breast cancer data above. Specifically, the algorithm examines the relative magnitudes of frequencies of observations and determines how every observation should be classified based on the predominant pattern in the data. Each observation is then evaluated according to this pattern and judged as either correctly or incorrectly classified. The results can be presented in a simple graph referred to as a “multigram.” To help students interpret multigrams, instructors should prepare at least two contrived sets of observations, like those reported in Table 2, showing clear patterns. Consider 60 weight observations, for instance, in which all of the women in therapy gain two or more pounds while all women in the control group lose weight or maintain their current weight (column labeled “Clear One” in Table 2). The resulting multigram is shown in Fig. 4. As can be seen, a multigram graphs the two frequency distributions side by side, and in this extreme example the two distributions do not overlap at all. A clear pattern is present in which the weight values can be separated into two groups. The Percent Correct Classification (PCC) index for these data is 100%, and the pattern clearly supports the effectiveness of the therapy. Every woman in the therapy group gained weight, and moreover gained more weight than every woman in the control group.

OPEN IN VIEWER

Fig. 4.

Multigram of clear pattern example. This pattern matches expectation with the individuals in the Therapy group gaining weight.

As another extreme example, consider data (“Clear Two” in Table 2) that generate the multigram in Fig. 5. Again, the 8-unit weight observations can be classified into the 2-unit group orderings with remarkable accuracy (PCC = 96.67%). The pattern in the multigram shows that: (1) all of the women who gained 2 or more pounds were in the Therapy group; (2) all who gained up to 1.9 pounds or lost as much as 6 pounds were in the Control group; and (3) most of the women who lost 6.1 or more pounds were in the Therapy group, contrary to expectation. Based on the relative frequencies both within the 8 weight units and the 2 group units, the algorithm determined that all of the observations should have fit this pattern and were therefore classified as such. As can be seen in Fig. 5, then, two observations (viz., two Control women who lost 6.1 or more pounds) were inconsistent with this overall pattern and were hence counted as misclassified by the algorithm even though they were in the Control group and lost relatively large amounts of weight. This example thus demonstrates that the algorithm works on the pattern of observations (data), not on the researcher's predicted pattern of results as in the breast cancer study above. Nine women in the Therapy group lost 6.1 or more pounds, which is entirely contrary to expectation; yet, the algorithm counted them as correctly classified. Only two women in the Control group lost this much weight, and all of the other women in the Therapy group lost more weight than all of the women in the Control group. Based on the clear separation between groups of observations, the algorithm determined that all women who lost 6.1 pounds or more should be classified as belonging to the Therapy group, which unfortunately contradicts expectation. The analysis is entirely ad hoc, however, making such outcomes possible. A high PCC index in the Build/Test Model analysis therefore only indicates that the observations formed a clear pattern. It does not mean that the pattern fits the scientist's expectation. The multigram must be examined – preferably in light of an integrated model – to decide whether the pattern is consistent with expectation or theory, and students will quickly realize that interpreting the effects (and causes) embodied in a multigram is clearly different from computing and interpreting statistical effects embodied in d and η².

OPEN IN VIEWER

Fig. 5.

Multigram of clear pattern example. This pattern does not match expectation perfectly, as nine women in the Therapy group lost weight. These nine women were classified as correct by the ad hoc algorithm used in the analysis because they were distinct from the other observations.

The extreme multigrams and their accompanying data in Table 2 can also be used to help students understand the chance value. Specifically, each student can be asked to shuffle randomly the weight change values (“Clear One” in Table 2) that generated the multigram in Fig. 4. Values can simply be changed in the OOM software, making certain the students randomize a large number of the 60 values across both groups. In this way, some weights for the women in the Therapy group will be given to women in the Control group and vice versa. Once the weights are randomly shuffled, the students run the binary Procrustes rotation and examine the multigram and PCC index. Fig. 6 shows an example multigram from the randomized data (labeled “Random”) in Table 2. The clear pattern in Fig. 4 has now been lost, and the PCC index for the randomized data, 61.67, is much smaller than the PCC index for the “Clear One” data (100%). The randomization test in OOM simply repeats this process a set number of times (e.g., 1000 trials) and determines the proportion of trials in which the PCC index equals or exceeds the original value. This proportion is the chance value, or c-value, and for the data in Figs. 4 and 5 it is less than .001, indicating that not one time in 1000 trials did the randomized data yield PCC indices as high as 100 or 96.67, respectively. A low c-value indicates that randomized versions of the observations do not readily produce a pattern as clear or discriminating as the one obtained for the actual data. It informs the students of the distinctiveness or unusualness of the observed PCC index and accompanying pattern, and it does so without any assumptions. Recall from above the list of assumptions necessary for the accuracy of the p value for the independent samples t test (e.g., normality of population distributions, homogeneous population variances, independence of observations). None of these assumptions are required for the randomization test (Manly, 2006), and students will consequently find the c-value to be concrete, simple, and intuitive in comparison to the traditional p value.

OPEN IN VIEWER

Fig. 6.

Multigram of randomized data with no clear pattern.

Turning students to the OOM results for the actual weight data in Table 2, the multigram in Fig. 7 reveals that a single discrimination point between the two frequency distributions cannot be made. Nonetheless, a pattern of bimodal, non-overlapping units is evident in the multigram, permitting a large number of the observations to be correctly and distinctively classified, PCC = 86.67, c-value < .001. The OOM results are as follows:

OPEN IN VIEWER

Fig. 7.

Multigram of weight gain data for Control and Therapy groups. This pattern partly matches expectation, but 11 women in the anorexia Therapy group lost weight and were classified correctly by the ad hoc algorithm because of their distinctiveness.

OPEN IN VIEWER

Classification Results
Conforming (Effect) Observations Classified to Target (Cause) Observations
Classifiable Observations:	60
Ambiguous Classifications:	0
Correct Classifications:	52
Percent Correct Classifications:	86.67
Randomization Results
Observed Percent Correct Classifications:	86.67
Number of Randomized Trials:	1000
Minimum Random Percent Correct:	28.33
Maximum Random Percent Correct:	78.33
Values >= Observed Percent Correct:	0
Model c-value: Less than (1/1000); that is, < 0.001

The OOM software has detected a pattern that classifies an overwhelming majority of the women correctly based on the observations themselves…but what is the meaning of the result? Because population parameters are not estimated, the three-valued logic hypotheses above are not relevant. Without an integrated model to guide their thinking, students must examine the multigram in Fig. 7 and interpret the pattern. As can be seen, it appears the effect of the therapy is dichotomous. Sixteen women have lost weight, which is a terrible outcome given they are anorexic, while 14 women have gained weight, which is a positive outcome. The Control group as well appears to be made up of two groups, those who lost approximately 1 to 8 pounds and those who gained approximately 1 to 4 pounds. Cohen's d and η² offered no information about these patterns, and to speak of d as indicating a “typical” standardized difference between women in the two groups is recognized as misleading. A majority of women in the Therapy group actually lost weight, contradicting the t test conclusion supportive of the efficacy of the therapy – a disaster for the anorexia therapy.

If the weight observations are represented as two units (–0.1 to −8.0, weight loss; 0.1 to 8.0, weight gain), then the meaning of the results for the women in the study becomes even more clear. As can be seen in the multigram in Fig. 8, the two groups of women appear highly similar with respect to the 2-unit weight observations; and again, a small majority of women in the anorexia therapy group actually lost weight. These 16 women in the Therapy group were therefore classified correctly by the algorithm, a result entirely opposite of expectation. This analysis also shows, however, no ability to discriminate clearly between the women in terms of simple gain or loss in weight, as indicated by a PCC index of 51.67 and c-value of 1. In every instance, randomized versions of the data yielded a PCC index of 51.67 or higher, indicating that the pattern in Fig. 8, although opposite of expectation, is not at all distinct.

OPEN IN VIEWER

Fig. 8.

Multigram of weight gain data for Control and Therapy groups. Weight gain has been ordered into two units, and the pattern is opposite of expectation with a slight majority of women in the anorexia Therapy group having lost weight.

Effect size indices accompanying results for most modern statistical analyses are understood to be aggregate-based or variable-based values that are difficult to interpret without the aid of established conventions, like Cohen's conventions. As such, they are not effects that are logically tied to the causes that inhere in the people in the study. Effect sizes are statistical, aggregate summaries of the data, and their magnitudes have no meaning for any given individual in the study. Through this particular class exercise, students realize that OOM analyses are not based on aggregate statistics. Not a single mean, median, or standard deviation has been invoked. The output across analyses is also highly similar, allowing students to focus on the primary information; viz., the patterns and PCC indices. This streamlining is possible because OOM reduces all data to a common binary form, referred to as deep structure, upon which the analyses are based. Moreover, the goal in OOM is not to estimate abstract population parameters, but to accurately classify the observations based on some a priori or ad hoc pattern (ideally, of course, an integrated model would be available for explaining the pattern). Accuracy is therefore the key judgment, and the PCC index is readily understood by students, other scientists, and laypersons as well. The c-value in OOM is also free of assumptions and relatively concrete compared to the traditional p value in null hypothesis significance testing, and is based on a firm foundation of reasoning dating to at least the 1960s (Winch & Campbell, 1969). Nonetheless, the inference to cause and effect is only aided by a high PCC index and low c-value. The basis and force of such an inference ultimately rest upon the structures and processes in an integrated model.

An Exercise in “Measurement”

Following modern psychometric theory, students are instructed at the beginning of this exercise that Need for Speed is a continuous latent variable that cannot be measured without error. More or less time can be spent on discussing the modern conceptualization of latent variables (e.g., see Bollen, 2002; Borsboom, 2008), but the pragmatic point is that multiple items must be constructed from which scores (values) can be obtained. These scores are expected to vary from person to person for two reasons: (1) variation due to true individual differences in Need for Speed, and (2) variation due to unknown factors (errors). The latter errors are often undifferentiated and expected to be normally distributed and independent. This is of course the simple classical true score model of psychometrics.

Eight items are written to include in the questionnaire, as shown in Fig. 9. Data are obtained from ratings on a Likert-type scale, i.e., arbitrarily numbered with values from 0 (highly disagree) to 6 (highly agree). The fictitious data for this exercise are reported in the Appendix. Students next conduct a standard item analysis, examining the inter-item correlations, corrected item-total correlations, and Cronbach's α. The results from this analysis will show that all pairs of items are positively correlated, and the corrected item-total correlations are all positive and greater than .45. Cronbach's α is also impressive, with a value of .83. Based on these results, the scale is said to be reliable with respect to internal consistency. Advanced students can also be required to factor analyze the data, the results of which will reveal a strong single factor on which each of the eight items loads positively (minimum loading = .58). Based on these results, the scale is said to possess some construct validity as it is judged to be unidimensional, which is consistent with the understanding of Need for Speed as a single, continuous latent variable.

OPEN IN VIEWER

Fig. 9.

Hypothetical Need for Speed questionnaire.

With sufficient reliability and evidence for a single dimension, students are taught to sum the item responses to generate a scale score that is allegedly more reliable and valid than any of the individual eight items. Advanced students can discuss the pros and cons of computing more complicated factor scores based on the results of the factor analysis. For the analyses below, however, simple sum scores will be used. Finally, as a segue to concurrent validity and perhaps multi-trait multi-method matrices, students can compare men and women (data are provided in the Appendix) on the Need for Speed sum scores using traditional statistics. The results for an independent samples t test are statistically significant at the .05 level (t₇₈ = 2.11, p = .038), with a medium effect size (d = 0.47, η² = 0.05) and relatively precise 95% confidence interval (0.21 ≤ μ₁ – μ₂ ≤7.04). As might be expected, the mean for men (M = 29.08, SD = 6.87) is higher than the mean for women (M = 25.45, SD = 8.40).

Having worked through these psychometric and statistical exercises, students are then challenged to critically examine their work and to transition to new ways of thinking. The first step is to ask students, “What exactly is Need for Speed?” Is it a less reasonable construct to investigate than depression, intelligence, need for cognition, perfectionism, gender, etc.? If it exists in human nature, exactly how does it exist? Is it reasonable to consider it to be a continuous dimension along which people can be measured in extremely precise units…even in theory? When discussing Need for Speed at this level, it is important to help students distinguish between qualities, things, and variables. A quality, speaking generally in the context of psychology, is something that exists within persons. When a student walks into a classroom, he or she recognizes various things, some of which are fellow students and some of which are objects used for instruction (tables, desks, marker boards, etc.). By differentiating between persons and objects, the student understands the substantial form, or “whatness,” of the things in the room. The student of course also realizes that some students are shorter than others; some have brown hair while others have black hair; some are more vocal than others, while others seem more reserved; and so on. These recognized differences inhere in the persons themselves and do not exist in and of themselves. They are what philosophical realists would refer to as “accidents.” In other words, a student walking into the classroom does not see “hair color,” “height,” “shyness,” etc. as things; rather, he or she sees fellow students who possess these qualities. The student sees Lisa who has black hair and Jeremy who has blond hair but never “hair color” as a thing or substantial form.

A variable is not a quality; rather, a variable is a product of human reason, as when a student states, “Let x equal scores on the Need for Speed questionnaire, and y equal measured height.” A variable is therefore a product of the human mind (subject) engaged with things and their qualities (substantial forms and accidents). Variables of course play an important role in science by providing formal representations of things and qualities that can be manipulated by reason alone. The important point being made here, however, is that the subject-object dialectic is important in philosophical realism and must be maintained. Too often statisticians and psychologists fall unintentionally into phenomenalism, which then creates a propitious context for remaining trapped in the abstract world of null hypothesis significance testing. Consider, e.g., David Howell's definition of variable in his excellent statistics textbook, “A variable is a property of an object or event that can take on different values” (2007, p. 3). This definition unfortunately conflates and confuses qualities (referred to as “properties”) that exist in the persons or objects of study, and variables that exist in the minds of the investigators.

While they certainly have not been without controversy over the centuries, Aristotle's ten categories of being may help students think more clearly about their research and about Need for Speed in the current exercise. The categories are listed, defined, and exemplified in Table 3 for a student named Sandy who completed the Need for Speed questionnaire in August of 2013 in the Personality Research Laboratory at Oklahoma State University (see Wallace, 1977; Studtmann, 2007). Examining the list, students will most likely regard Need for Speed as a quality, particularly a habit, which for Aristotle was an enduring tendency to act in certain ways to consistently achieve a particular type of end. A virtue such as justice, e.g., can be said to be a habit if the choices and actions of a person repeatedly result in right and just outcomes. Need for Speed may similarly be thought of as such a tendency. The great difficulty over the centuries, however, has been whether or not some qualities are “rooted in quantity” (Wallace, 1977, p. 30). The rulerlike appearance of the Likert-type scale with its numbered values and the sum scores for the Need for Speed questionnaire suggest the quality can be understood as a continuous quantity; what St. Thomas Aquinas referred to as a “quantitative quality” or quantitas virtutis (see Crowley, 1996; Grice, 2011, Chapter 8; Echavarría, 2013). Absolutely no evidence, however, has been presented to support this claim. The students may appeal to the excellent results from the item analysis and factor analysis of the Need for Speed questionnaire, but as Michell (1999, 2008, 2011) has made clear, these analyses (and structural equation and item response models as well) assume continuous quantities, they do not provide evidence for quantitative structure.

OPEN IN VIEWER

TABLE 3

Aristotle's Ten Categories of Being

Category	Description	Example
Substance	“What is it?”; An existing thing	A person, Sandy.
Quantity	“How much?”; Continuous or Discrete	5′ 4″ tall (continuous); 2 eyes, 2 ears (discrete).
Quality	Habits, Dispositions, Powers, Sensible Qualities, Figure and Shape	Blue eyes; Consistently courteous and kind; Intelligent; Petit stature.
Relation	Relation of one substance to another	Daughter of Janet Johnson; participant helping experimenter.
Action	“To do”; Change produced by a substance	Read and understood instructions and statements, then wrote numbers as responses.
Passion	Changes occurring within a substance	Grew angry while filling out questionnaire.
Location	“Where”; Place	Personality Research Laboratory.
Position	Posture of agent	Walking (to lab), Sitting (when filling out questionnaire).
Time	“When”	12:30 PM, August 20th, 2013.
Vestition	“To have”; Possession	Sandy was clothed in a blue dress and wore a bow in her hair.

An effective way to prompt students to think critically about the continuity assumption, and indeed the assumptions of the classical true score model of psychometrics, is to consider scores from two respondents on Items 5, 6, and 7 of the Need for Speed questionnaire, as shown here:

OPEN IN VIEWER

John	Luke
3	0	I've always taken quick showers/baths (I5)
3	0	I like to watch car races (I6)
0	6	Red is my favorite color for a car (I7)
6	6	Sum Score

Both John and Luke received sum scores of 6 on these three items. Are the students confident that both individuals possess the same amount of Need for Speed? If John were to have received a score of 12, would the students be confident in stating that he possessed twice the amount of Need for Speed as Luke? Can Need for Speed be divided into smaller and smaller equal interval units of measure using such self report methods? These are all questions relevant to addressing whether or not Need for Speed is structured as a continuous quantity. In other words, can it be analogously understood as a ruler, and is the process of summing here truly equivalent to summing up equal units from a single ruler? This last question can be approached by asking students to debate whether or not the items are truly interchangeable representations of Need for Speed. Do they essentially mean the same thing? The correlations between these three items are statistically significant at the .05 level and positive: r₅₆ = .46, r₅₇ = .43, and r₆₇ = .27. Do the magnitudes of these correlations support the argument that the persons completing the questionnaire, like John and Luke, regarded the items as having essentially the same meaning?

As discussed above, interpreting aggregate statistics can be difficult and misleading. The same pattern analysis in the OOM software used for the breast cancer data can be used here to obtain a clearer picture of the data. Specifically, the observations on pairs of the three selected items should form clear patterns in which their labeled units (0, 1, 2, etc.) are matched. Figure 10, however, tells a different story. Very few observations fit the expected patterns as the largest PCC index was only 41.25 for Items 5 and 7. The other two PCC indices were below 27%. These results show clearly that each person completing the three selected items rarely used the same points on the Likert-type rating scale. Examining all eight items reveals similarly low PCC values.

OPEN IN VIEWER

Fig. 10.

Patterns of observations for three Need for Speed Items.

The OOM software employs a simple algorithm that can be used to quickly examine expected unit-to-unit patterns like that shown in Fig. 10. Such unit-to-unit patterns are commonly employed by psychologists, such as when examining inter-rater consistency. Results from the OOM Matching Analysis shown in Table 4 indicate clearly a general lack of consistency in judgments across the eight items on the Likert-type scale. The highest PCC index was 58.75 and the lowest was 13.75. Most values were below 30%, and the total PCC value across all persons and all eight items was only 28.97, although the c-value (0.10) indicated this outcome was unusual. These results and examination of John's and Luke's individual sum scores shed light on the dubiousness of assuming a single continuously structured quality somehow underlying the eight Need for Speed questionnaire items that moreover acts as a cause of the individuals' responses to the items.

OPEN IN VIEWER

TABLE 4

Need For Speed Matching Results From OOM Software

	Percent Matches (PCC)
	I1	I2	I3	I4	I5	I6	I7	I8
I1	100.00
I2	56.25	100.00
I3	42.50	21.25	100.00
I4	36.25	22.50	30.00	100.00
I5	18.75	15.00	28.75	27.50	100.00
I6	33.75	25.00	37.50	27.50	26.25	100.00
I7	23.75	16.25	16.25	41.25	41.25	13.75	100.00
I8	26.25	21.25	28.75	28.75	58.75	18.75	27.50	100.00

Further analysis in OOM of the individual item and sum score observations in relation to gender (male / female) is also instructive. Recall that an independent samples t test revealed a statistically significant difference between men and women on the Need for Speed sum score, with men yielding a higher mean than women. Analysis in OOM reveals a different story. As can be seen in the multigram in Fig. 11, the observations for men and women overlapped a great deal across the 49 units (possible sum scores ranged from 0 to 48). Sum scores for four women were atypically low, which may have dragged the mean for women down. Otherwise, no clear pattern emerged differentiating men and women with regard to the observed sum scores. Interestingly, the PCC index for this analysis was mildly impressive at 70.00, but the c-value (.86) indicated that a value at least this large was readily obtained from randomized versions of the same observations. Even without the c-value, the lack of any clear visible pattern in the multigram is likely sufficient to deter putting forth a causal argument in this example.

OPEN IN VIEWER

Fig. 11.

Multigram of sum scores from Need for Speed inventory and gender.

Students will find analysis of the individual items, however, to be more instructive. As can be seen in the multigrams and PCC indices in Fig. 12, a relatively clear pattern emerged when attempting to conform the observations for Item 7 (“Red is my favorite color for a car”) to gender (male / female). Most men (63%) were observed in the 4, 5, or 6 units; whereas most women (58%) were observed in the 0, 1, or 3 units. The PCC index was fairly high (70%) and the c-value very low (0.003). Several other items yielded PCC indices close to 70%, but the patterns were not strong, visually speaking, nor were the c-values very low. With more men agreeing to the statement “Red is my favorite color for a car” than women, does this in fact mean that these men were expressing a Need for Speed? If every item is an equivalent ruler for Need for Speed, why did the same differences in proportions (i.e., the same pattern in the multigrams) of men and women not emerge for the other seven statements? These are the questions students must grapple with in an attempt to finally come to grips with the measurement problems in psychology and to find a viable way forward without letting methods trump metaphysics.

OPEN IN VIEWER

Fig. 12.

Multigrams, PCC indices, and c-values for eight Need for Speed items.

Finally, conducting independent samples t tests on the eight individual items also reveals the dilemma. Only Item 7 yields a statistically significant difference between men and women. Aggregating the eight items essentially buried the salient difference between the two sexes on Item 7 while creating the illusion that important, “statistically significant,” differences were assured on the other items as well since they measured the same quality.

This example and the anorexia study above demonstrate why a motto in Observation Oriented Modeling is “aggregation often leads to obfuscation.” To be clear, aggregation is meant to express the standard methods of computing means, variances, and covariances and all of the complex statistical methods based on these statistics. Aggregation may also refer to the simple tallying of responses or behaviors, a procedure common in the OOM software. The next generation must finally shed the “physics envy” and phenomenalism of modern psychologists who view the whole of science as continuous, quantitative measurement of qualities that are not truly knowable. Counting, or working with discrete quantities or units, underlies continuous measurement; hence, there is no shame or second class citizenship to be feared in dealing mainly with discrete quantities or speaking in terms of structures and processes rather than alleged continuous variables. It can also be noted that in Table 3, quantity is only one of Aristotle's ten categories of being.

With regard to Need for Speed, then, what alternatives might be pursued? Students can be encouraged to develop behavioral checklists, obtained via self-report or observation, as well as single-item approaches. Medical doctors often employ checklists to aid in diagnosis and treatment, and Barrett (2008) offers the Violence Risk Assessment Guide as a successful example checklist (accurately predicting violent recidivism at 72%) developed without the use of latent variable models or other sophisticated statistical models. With regard to single-item self-report approaches, they have been shown to be adequate replacements for multiple-item questionnaires in a number of contexts (e.g., see Nagy, 2002; Bergkvist & Rossiter, 2007). Rather than “measuring” extraversion with ten items, for instance, a single well-phrased item (Woods & Hampson, 2005) or a vignette can be developed:

The person reading the vignette can be asked to affirm or deny the description as applying to himself or herself, thus avoiding a Likert-type or other form of rating scale. Using vignettes, in particular, requires students to think more deeply and concretely about the meanings of the qualities they are attempting to research. Behavioral checklists and single items will also yield units of observation more readily incorporated into integrated models because students will have to take greater responsibility in articulating the exact meaning of any numerical values they obtain.