This proof-of-concept investigation describes a computer-based approach for deriving the knowledge structure of individuals and of groups from their written essays, and considers the convergent criterion-related validity of the computer-based scores relative to human rater essay scores and multiple-choice test scores. After completing a classroom-based course, undergraduate participants in a sophomore-level management course completed a 100-item multiple-choice final examination and then answered an extended-response essay question comparing four management theories. The essays were quantified with ALA-Reader software applying both sentence-wise and linear lexical aggregate approaches, and then analyzed with Pathfinder KNOT software. The linear aggregate approach was a better measure of essay content structure relative to the sentence-wise approach, with significant Spearman correlations of 0.60 and 0.45 with the human rater essay scores. The group network representations of low and high performing students were reasonable and straightforward to interpret, with the high group being more similar to the expert, and the low and high groups more similar to each other than to the expert. Suggestions for further research are provided.
Get full access to this article
View all access options for this article.
References
1.
AndersonJ. R. (1983). The architecture of cognition.Cambridge, MA: Harvard University.
2.
AndersonR. C. (1984). Some reflections on the acquisition of knowledge. Educational Researcher, 13(10), 5–10.
3.
BarrittL.StockP. L., & ClarkF. (1986). Researching practice: Evaluating assessment essays. College Composition and Communication, 37, 315–327.
4.
BiggsJ. (1992). A qualitative approach to grading students. Higher Education Research and Development Society of Australasia News, 14(3), 3–6
5.
BrooksL. W.DansereauD. F.SpurlinJ. E., & HolleyC. D. (1983). Effects of headings on text processing. Journal of Educational Psychology, 75, 292–302.
6.
ChiM. T. H.GlaserR., & FarrM. J. (1988). The nature of expertise.Hillsdale, NJ: Erlbaum.
7.
ChiM. T. H.GlaserR., & ReesE. (1982). Expertise in problem solving. In SternbergR. S. (Ed.), Advances in the psychology of human intelligence (pp. 1–75). Hillsdale, NJ: Erlbaum.
DeeseJ. (1965). The structure of associations in language and thought.Baltimore, MD: The Johns Hopkins Press.
10.
ElmanJ. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 71–99.
11.
GoldsmithT. E., & JohnsonP. J. (1990). A structural assessment of classroom learning. In SchvaneveldtR. W. (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 241–253). Norwood, NJ: Ablex Publishing Corporation.
12.
GoldsmithT. E.JohnsonP. J., & ActonW. H. (1991). Assessing knowledge structure. Journal of Educational Psychology, 83, 88–96.
13.
GonzalvoP.CanasJ. J., & BajoM. (1994). Structural representations in knowledge acquisition. Journal of Educational Psychology, 86, 601–616.
14.
JonassenD. H.BeissnerK., & YacciM. (1993). Knowledge structure: Techniques for representing, conveying, and acquiring knowledge structure.Hillsdale, NJ: Lawrence Erlbaum and Associates.
15.
JonassenD. H., & WangS. (1992). Acquiring structural knowledge from seman-tically structured hypertext. Proceedings of Selected Research and Development Presentations at the 14th Annual Convention of the Association for Educational Communications and Technology. (ERIC Document Reproduction Service No. ED 348 000).
16.
KoulR.ClarianaR. B., & SalehiR. (2005). Comparing several human and computer based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32(3), 261–273.
17.
LandauerT. K.FoltzP. W., & LahamD. (1998). Introduction to latent semantic analysis. Discourse Processes, 25, 259–284.
18.
LandauerT. K.LahamD.RehderR., & SchreinerM. E. (1997). How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. In ShaftoM. G. & LangleyP. (Eds.), Proceedings of the 19th annual meeting of the Cognitive Science Society (pp. 412–417). Mahwah, NJ: Erlbaum.
19.
LomaskM.BaronJ.B.GreigJ., & HarrisonC.(1992, March) ConnMap: Connecticut's use of concept mapping to assess the structure of students' knowledge of science. Paper presented at the annual meeting of the National Association of Research in Science Teaching, in Cambridge, MA.
20.
LuhnH. P. (1959). Keyword-in-context index for technical literature (KWIC index). (Technical report RC-127, IBM Corporation, Advanced Systems Development Division). Reprinted in SchultzC. K. (Ed.). (1968). H. P. Luhn: Pioneer of information science: Selected works (pp. 227–235). New York: Spartan Books.
21.
National Research Council. (2001). Knowing what students know: The science and design of educational assessment.Washington, DC: National Academies Press.
22.
NitkoA. J. (1996). Educational assessment of students.Englewood Cliffs, NJ: Merrill.
23.
NovakJ. D. (2003). The promise of new ideas and new technology for improving teaching and learning. Cell Biology Education, 2, 122–132.
24.
RudnerL. M., & LiangT. (2002). Automated essay scoring using Bayes' theorem. Journal of Technology, Learning, and Assessment, 1 (2). Retrieved May 5, 2005, from http://www.jtla.org
25.
RumelhartD., & McClellandJ. (1986). Parallel distributed processing (Vols. I and II). Cambridge, MA: MIT Press.
26.
SchvaneveldtR. W. (1990). Pathfinder associative networks: Studies in knowledge organization.Norwood, NJ: Ablex Publishing Corporation.
27.
ShavelsonR. J. (1972). Some aspects of the correspondence between content structure and cognitive structure in physics instruction. Journal of Educational Psychology, 63, 225–234.
28.
ShavelsonR. J. (1974). Some methods for examining content structure and cognitive structure in instruction. Educational Psychologist, 11, 110–122.
29.
ShavelsonR. J.LangH., & LewinB. (1994). On concept maps as potential “authentic” assessments in science (CSE Tech. Rep. No. 388). Los Angeles, CA: University of California, Center for Research on Evaluation, Standards, and Student Testing (CREST).
TaberK. S. (2003). Examining structure and context—Questioning the nature and purpose of summative assessment. Seminar presentation to Cambridge International Examinations, University of Cambridge Local Examinations Syndicate, July 2003. Retrieved May 04, 2005 from http://www.leeds.ac.uk/educol/documents/00003134.htm
32.
ValentiS.NeriF., & CucchiarelliA. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education, 2, 319–330. Retrieved May 14, 2005 from http://jite.org/documents/Vol2/v2p319–330–30.pdf
33.
YagerR. E. (1983). The importance of terminology in teaching K-12 science. Journal of Research in Science Teaching, 20(6), 577–588.
34.
ZajchowskiR., & MartinJ. (1993). Differences in the problem solving of stronger and weaker novices in physics: Knowledge, strategies, or knowledge structure?Journal of Research in Science Teaching, 30(5), 459–470.
