Studies employing multiple methods of cognitive assessment have often found low convergent validity, especially between endorsement (e.g., a checklist or questionnaire) and production e.g., think aloud or thought listing) methods. These results raise serious questions about the construct validity of measures receiving extensive use in cognitive behavior therapy research and practice. Five possible explanations for low convergent validity of cognitive assessments are elaborated: unreliability of production measures due to insufficient aggregation, noncomparable stimulus sampling across measures, use of different judges on each measure to make subjective ratings, invalidity of endorsement measures resulting from the nomothetic assumption that the same cognitions apply equally well to everyone, and invalidity resulting from arbitrary decisions about response formats during test construction. Suggestions regarding how these explanations might be tested are described, and in the case of the last explanation an original study illustrating such a research strategy is presented.
Get full access to this article
View all access options for this article.
References
1.
Ahrens, A. H. , & Haaga, D.A.F. (1993). The specificity of attributional style and expectations to positive and negative affectivity, depression, and anxiety. Cognitive Therapy and Research, 17, 83-98.
2.
Barber, J. P. , & DeRubeis, R. J. (1992). The Ways of Responding: A scale to assess compensatory skills taught in cognitive therapy. Behavioral Assessment, 14, 93-115.
3.
Baumeister, R. F. , & Tice, D. M. (1988). Metatraits. Journal of Personality, 56, 571-598.
4.
Bellack, A. S. (1983). Recurrent problems in the behavioral assessment of social skill. Behaviour Research and Therapy, 9, 448-461.
5.
Bem, D. J. , & Allen, A. (1974). On predicting some of the people some of the time: The search for cross-situational consistencies in behavior. Psychological Review, 81, 506-520.
6.
Blackwell, R. T. , Galassi, J. P., Galassi, M. D., & Watson, T. E. (1985). Are cognitive assessment methods equal? A comparison of think aloud and thought listing. Cognitive Therapy and Research, 9, 399-413.
7.
Bradburn, N. M. , Rips, L. J., & Shevell, S. K. (1987). Answering autobiographical questions: The impact of memory and inference on surveys. Science, 236, 157-161.
8.
Burgess, E. , & Haaga, D.A.F. (1994). The Positive Automatic Thoughts Questionnaire (ATQ-P) and the Automatic Thoughts Questionnaire-Revised (ATQ-RP): Equivalent measures of positive thinking?Cognitive Therapy and Research, 18, 15-23.
9.
Cacioppo, J. T. , & Petty, R. E. (1981). Social psychological procedures for cognitive response assessment: The thought-listing technique. In T. V. Merluzzi, C. R. Glass, & M. Genest (Eds.), Cognitive assessment (pp. 309-342). New York: Guilford.
10.
Campbell, D. T. , & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
11.
Chaplin, W. F. (1991). The next generator of moderator research in personality psychology. Journal of Personality, 59, 143-178.
12.
Chiauzzi, E. J. , Heimberg, R. G., Becker, R. E., & Gansler, D. (1985). Personalized versus standard role plays in the assessment of depressed patients’social skill. Journal of Psychopathology and Behavioral Assessment, 7, 121-133.
13.
Clark, D. A. (1988). The validity of measures of cognition: A review of the literature. Cognitive Therapy and Research, 12, 1-20.
14.
Coyne, J. C. (1989). Thinking postcognitively about depression. In A. Freeman, K. M. Simon, L. E. Beutler, & H. Arkowitz (Eds.), Comprehensive handbook of cognitive therapy (pp. 227-244). New York: Plenum.
15.
Csikszentmihalyi, M. , & Larson, R. (1987). Validity and reliability of the experience sampling method. Journal of Nervous and Mental Disease, 175, 526-536.
16.
Davison, G. C. , Feldman, P. M., & Osborn, C. E. (1984). Articulated thoughts, irrational beliefs, and fear of negative evaluation. Cognitive Therapy and Research, 8, 349-362.
17.
Davison, G. C. , Navarre, S. G., & Vogel, R. S. (1995). The articulated thoughts in simulated situations paradigm: A think-aloud approach to cognitive assessment. Current Directions in Psychological Science, 4, 29-33.
18.
Davison, G. C. , Robins, C., & Johnson, M. K. (1983). Articulated thoughts during simulated situations: A paradigm for studying cognition in emotion and behavior. Cognitive Therapy and Research, 7, 17-40.
19.
Davison, G. C. , & Zighelboim, V. (1987). Irrational beliefs in the articulated thoughts of college students with social anxiety. Journal of Rational-Emotive Therapy, 5, 238-254.
20.
Dohrenwend, B. S. , Krasnoff, L., Askenasy, A. R., & Dohrenwend, B. P. (1978). Exemplification of a method for scaling life events: The PERI Life Events Scale. Journal of Health and Social Behavior, 19, 205-229.
21.
Epstein, S. (1979). The stability of behavior: I. On predicting most of the people much of the time. Journal of Personality and Social Psychology, 37, 1097-1126.
22.
Epstein, S. (1983). Aggregation and beyond: Some basic issues in the prediction of behavior. Journal of Personality, 51, 360-391.
23.
Exner, J. E. (1978). The Rorschach: A comprehensive system: Vol. 2. Current research and advanced interpretation. New York: Wiley.
24.
Fletcher, G.J.O. , Danilovics, P., Fernandez, G., Peterson, D., & Reeder, D. (1986). Attributional complexity: An individual differences measure. Journal of Personality and Social Psychology, 51, 875-884.
25.
Glass, C. R. (1993). A little more about cognitive assessment. Journal of Counseling and Development, 71, 546-548.
26.
Glass, C. R. , & Arnkoff, D. B. (1982). Think cognitively: Selected issues in cognitive assessment and therapy. In P. C. Kendall (Ed.), Advances in cognitive-behavioral research and therapy, vol. 1 (pp. 35-71). New York: Academic Press.
27.
Glass, C. R. , & Arnkoff, D. B. (1994). Validity issues in self-statement measures of social phobia and social anxiety. Behaviour Research and Therapy, 32, 255-267.
28.
Glass, C. R. , Merluzzi, T. V., Biever, J. L., & Larsen, K. H. (1982). Cognitive assessment of social anxiety: Development and validation of a self-statement questionnaire. Cognitive Therapy and Research, 6, 35-55.
29.
Haaga, D.A.F. (1989). Articulated thoughts and endorsement procedures for cognitive assessment in the prediction of smoking relapse. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 1, 112-117.
30.
Haaga, D.A.F. , Ahrens, A. H., Schulman, P., Seligman, M.E.P., DeRubeis, R. J., & Minarik, M. L. (1995). Metatraits and cognitive assessment: Application to attributional style and depressive symptoms. Cognitive Therapy and Research, 19, 121-142.
31.
Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior Analysis, 10, 103-116.
32.
Heimberg, R. G. , Bruch, M. A., Hope, D. A., & Dombeck, M. (1990). Evaluating the states of mind model: Comparison to an alternative model and effects of method of cognitive assessment. Cognitive Therapy and Research, 14, 543-557.
33.
Heimberg, R. G. , Nyman, D., & O’Brien, G. T. (1987). Assessing variations of the thought-listing technique: Effects of instructions, stimulus intensity, stimulus modality, and scoring procedures. Cognitive Therapy and Research, 11, 13-24.
34.
Kanazawa, S. (1992). Outcome or expectancy? Antecedent of spontaneous causal attribution. Personality and Social Psychology Bulletin, 18, 659-668.
35.
Kanner, A. D. , Coyne, J. C., Schaefer, C., & Lazarus, R. S. (1981). Comparison of two modes of stress measurement: Daily hassles and uplifts versus major life events. Journal of Behavioral Medicine, 4, 1-26.
36.
Kastner, M. (1986). Pragmatic validity to be considered for the construction and application of psychological questionnaires. In A. Angleitner & J. S. Wiggins (Eds.), Personality assessment via questionnaires: Current issues in theory and measurement (pp. 49-60). New York: Springer-Verlag.
37.
Kendall, P. C. , & Hollon, S. D. (1981). Assessing self-referent speech: Methods in the measurement of self-statements. In P. C. Kendall & S. D. Hollon (Eds.), Assessment strategies for cognitive-behavioral interventions. New York: Academic Press.
38.
Kendall, P. C. , & Korgeski, G. P. (1979). Assessment and cognitive-behavioral interventions. Cognitive Therapy and Research, 3, 1-21.
39.
Kern, J. M. (1991). An evaluation of a novel role-play methodology: The standardized idiographic approach. Behavior Therapy, 22, 13-29.
40.
Krantz, S. E. , & Rude, S. (1984). Depressive attributions: Selection of different causes or assignment of dimensional meanings?Journal of Personality and Social Psychology, 47, 193-203.
41.
Kuiper, N. A. (1978). Depression and causal attributions for success and failure. Journal of Personality and Social Psychology, 38, 236-246.
42.
Lanning, K. (1988). Individual differences in scalability: An alternative conception for personality theory and measurement. Journal of Personality and Social Psychology, 55, 142-148.
43.
Loftus, E. F. , & Fathi, D. C. (1985). Retrieving multiple autobiographical memories. Social Cognition, 3, 280-295.
44.
Loftus, E. F. , Fienberg, S. E., & Tanur, J. M. (1985). Cognitive psychology meets the national survey. American Psychologist, 40, 175-180.
45.
Malouff, J. M. , & Schutte, N. S. (1986). Development and validation of a measure of irrational belief. Journal of Consulting and Clinical Psychology, 54, 860-862.
46.
Miranda, J. , & Persons, J. B. (1988). Dysfunctional attitudes are mood-state dependent. Journal of Abnormal Psychology, 97, 76-79.
47.
Peterson, C. , Bettes, B. A., & Seligman, M.E.P. (1985). Depressive symptoms and unprompted causal attributions: Content analysis. Behaviour Research and Therapy, 23, 379-382.
48.
Peterson, C. , Luborsky, L., & Seligman, M.E.P. (1983). Attributions and depressive mood shifts: A case study using the symptom-context method. Journal of Abnormal Psychology, 92, 96-103.
49.
Peterson, C. , Semmel, A., von Baeyer, C., Abramson, L. Y., Metalsky, G. I., & Seligman, M.E.P. (1982). The Attributional Style Questionnaire. Cognitive Therapy and Research, 6, 287-299.
50.
Peterson, C. , & Villanova, P. (1988). An expanded attributional style questionnaire. Journal of Abnormal Psychology, 97, 87-89.
51.
Reise, S. P. , & Waller, N. G. (1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65, 143-151.
52.
Robins, C. J. , Hayes, A. M., Block, P., Kramer, R. J., & Villena, M. (1995). Interpersonal and achievement concerns and the depressive vulnerability and symptom specificity hypotheses: A prospective study. Cognitive Therapy and Research, 19, 1-20.
53.
Segal, Z. V. , & Shaw, B. F. (1988). Cognitive assessment: Issues and methods. In K. S. Dobson (Ed.), Handbook of cognitive-behavioral therapies (pp. 39-81). New York: Guilford.
54.
Swendsen, J. D. (1997). Anxiety, depression, and their comorbidity: An experience sampling test of the helplessness-hopelessness theory. Cognitive Therapy and Research, 21, 97-114.
55.
Tarico, V. S. , Van Velzen, D. R., & Altmaier, E. M. (1986). Comparison of thought-listing rating methods. Journal of Counseling Psychology, 33, 81-83.
56.
Whitten, W. B. II , & Leonard, J. M. (1981). Directed search through autobiographical memory. Memory and Cognition, 9, 566-579.
57.
Zuckerman, M. , Bernieri, F., Koestner, R., & Rosenthal, R. (1989). To predict some of the people some of the time: In search of moderators. Journal of Personality and Social Psychology, 57, 279-293.
