In this paper, we review research on the perception and comprehension of synthetic speech produced by rule. We discuss the difficulties that synthetic speech causes for the listener and the evidence that the immediate result of those difficulties is a delay in the point at which words are recognized. We then argue that this delay in processing affects not only lexical access but also comprehension processes. We consider the mechanisms by which the comprehension system adjusts to this delay, the resulting costs to higher level comprehension processes, and the changes that occur in the language processing system as its familiarity with synthetic speech increases. Based on the framework we have developed, we suggest several directions for future research on the comprehension of synthetic speech.
Allen, J., Hunnicutt, M.S., and Klatt, D. (1987). From Text to Speech: The MITalk System.Cambridge, UK: Cambridge University Press.
2.
Altmann, G.T.M. (ed.) (1990). Cognitive Models of Speech Processing: Psycholinguistic and Computational Perspectives.Cambridge, MA: MIT Press.
3.
Auberge, V. (1992). Developing a structured lexicon for synthesis of prosody. In G. Bailly, C. Benoit, and T.R. Sawallis (eds.), Talking Machines: Theories, Models, and Designs (pp. 307–321). Amsterdam: North-Holland.
4.
Bard, E.G., Shillcock, R.C., and Altmann, G.T.M. (1988). The recognition of words after their acoustic offsets in spontaneous speech: Effects of subsequent context. Perception & Psychophysics, 44, 395–408.
5.
Balota, D.A., Flores d'Arcais, G., and Rayner, K. (eds.) (1990). Comprehension Processes in Reading.Hillsdale, NJ: Erlbaum.
6.
Benoit, C. (1990). An intelligibility test using semantically unpredictable sentences: Towards the quantification of linguistic complexity. Speech Communication, 9, 293–304.
7.
Bolinger, D.L. (1972). Accent is predictable (if you're a mindreader). Language, 48, 633–644.
8.
Clark, J.F., Dermody, P., and Palethorpe, S. (1985). Cue enhancement by stimulus repetition: Natural and synthetic speech comparisons. Journal of the Acoustical Society of America, 78, 458–462.
9.
Coltheart, M. (ed.) (1987). Attention and Performance XII: The Psychology of Reading.Hove, UK: Erlbaum.
10.
Cooper, F.S., Delattre, P.C., Liberman, A.M., Borst, J.M., and Gerstman, L.J. (1952). Some experiments on the perception of synthetic speech sounds. Journal of the Acoustical Society of America, 24, 597–606.
11.
Cooper, F.S., Liberman, A.M., and Borst, J.M. (1951). The interconversion of audible and visible patterns as a basis for research in the perception of speech. Proceedings of the National Academy of Sciences, 37, 318–325.
12.
Cutler, A. (1976). Phoneme-monitoring reaction time as a function of preceding intonation contour. Perception & Psychophysics, 20, 55–60.
13.
Duffy, S.A., Henderson, J.M., and Morris, R.K. (1989). Semantic facilitation of lexical access during sentence processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 791–801.
14.
Duffy, S.A., and Pisoni, D.B. (1991). Effects of sentence context on the signal duration required to identify natural and synthetic words.Research on Speech Perception Progress Report No. 17 (pp. 341–354). Bloomington, IN: Indiana University.
15.
Duffy, S.A., and Rayner, K. (1990). Eye movements and anaphor resolution: Effects of antecedent typicality and distance. Language and Speech, 33, 103–119.
16.
Fairbanks, B., Guttman, N., and Miron, M.S. (1957). The effects of time compression upon the comprehension of connected speech. Journal of Speech and Hearing Disorders, 22, 10–19.
17.
Flege, J. (1988). The production and perception of foreign language speech sounds. In H. Winitz (ed.), Human Communication and Its Disorders. A Review 1988.Norwood, NJ: Ablex.
18.
Fodor, J.A. (1983). The Modularity of Mind.Cambridge, MA: MIT Press.
19.
Forster, K.I. (1979). Levels of processing and the structure of the language processor. In W.E. Cooper and E. Walker (eds.), Sentence Processing: Psycholinguistic Studies Presented to Merrill Garrett (pp. 27–85). Hillsdale, NJ: Erlbaum.
20.
Fowler, C.A., and Housum, J. (1987). Talkers' signaling of “new” and “old” words in speech and listeners' perception and use of the distinction. Journal of Memory and Language, 26, 489–504.
21.
Frauenfelder, U.H., and Tyler, L.K. (eds.) (1987). Spoken Word Recognition.Cambridge, MA: MIT Press.
22.
Ganong, W.F. (1980). Phonetic categorization in auditory word perception. Journal of Experimental Psychology: Human Perception and Performance, 6, 110–125.
23.
Gernsbacher, M.A., and Faust, M.E. (1991). The mechanism of suppression: A component of general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17, 245–262.
24.
Gernsbacher, M.A., Varner, K.R., and Faust, M.E. (1990). Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 430–445.
25.
Gladstein, L., Membrino, M., and Flege, J. (1983). The Effect of Context on Americans' Perceptions of Word-Final Stop Devoicing by Vietnamese L2 Learners. Unpublished manuscript. Department of Linguistics, Georgetown University, Washington, DC.
26.
Goldinger, S.D., Pisoni, D.B., and Logan, J.S. (1991). On the nature of talker variability effects on recall of spoken word lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17, 152–162.
27.
Graesser, A.C., Hoffman, N.L., and Clark, L.F. (1980). Structural components in reading limes. Journal of Verbal Learning and Verbal Behavior, 19, 135–151.
28.
Greenspan, S.L., Nusbaum, H.C., and Pisoni, D.B. (1988). Perceptual learning of synthetic speech produced by rule. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 421–433.
29.
Grosjean, F. (1980). Spoken word recognition processes and the gating paradigm. Perception & Psychophysics, 28, 267–283.
30.
Grosjean, F. (1985). The recognition of words after their acoustic offset: Evidence and implications. Perception & Psychophysics, 38, 299–310.
31.
Haberlandt, K. (1984). Components of sentence and word reading times. In. D.E. Kieras and M.A. Just (eds.), New Methods in Reading Comprehension Research (pp. 219–252). Hills-dale, NJ: Erlbaum.
32.
Hersh, H.M., and Tartaglia, L. (1983). Understanding Synthetic Speech. User Research Group, Corporate Research and Architecture. Maynard MA: Digital Equipment Corp.
33.
Hirschberg, J. (1992). Using discourse context to guide pitch accent decisions in synthetic speech. In G. Bailly, C. Benoit, and T.R. Sawallis (eds.), Talking Machines: Theories, Models, and Designs (pp. 367–376). Amsterdam: North-Holland.
34.
Hoover, J., Reichle, J., VAN Tasell, D., and Cole, D. (1987). The intelligibility of synthesized speech: Echo II versus Votrax. Journal of Speech and Hearing Research, 30, 425–431.
35.
Ingemann, F. (1978). Speech synthesis by rule using the FOVE program.Haskins Laboratories Status Report on Speech Research, SR-54 (pp. 165–173). New Haven, CT: Haskins Laboratories.
36.
Jenkins, J.J., and Franklin, L.D. (1982). Recall of passages of synthetic speech. Bulletin of the Psychonomic Society, 20, 203–206.
37.
Just, M.A., and Carpenter, P.A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329–354.
38.
Just, M.A., and Carpenter, P.A. (1987). The Psychology of Reading and Language Comprehension.Boston: Allyn and Bacon.
39.
Kahneman, D. (1973). Attention and Effort.Englewood Cliffs, NJ: Prentice-Hall.
40.
Kilborn, K. (1989). Sentence processing in a second language: The timing of transfer. Language and Speech, 32, 1–23.
41.
Kilborn, K. (1992). On-line integration of grammatical information in a second language. In R.J. Harris (ed.), Cognitive Processing in Bilinguals (pp. 337–350). Amsterdam: North-Holland.
42.
Kintsch, W., and VAN Dijk, T.A. (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363–394.
43.
Klatt, D.H. (1987). Review of text-to-speech conversion for English. Journal of the Acoustical Society of America, 82, 737–793.
44.
Larkey, L., and Danly, M. (1983). Fundamental frequency and sentence comprehension. MIT Speech Group Working Papers, Vol. II (pp. 25–39). Cambridge, MA: Research Laboratory of Electronics, MIT.
45.
Liberman, A.M., Ingemann, F., Lisker, L., Delattre, P.C., and Cooper, F.S. (1959). Minimal rules for synthesizing speech. Journal of the Acoustical Society of America, 31, 1490–1499.
46.
Logan, J.S., Greene, B.G., and Pisoni, D.B. (1989). Segmental intelligibility of synthetic speech produced by rule. Journal of the Acoustical Society of America, 86, 566–581.
47.
Luce, P.A. (1981). Comprehension of fluent synthetic speech produced by rule.Research on Speech Perception Progress Report No. 7 (pp. 229–242). Bloomington, IN: Indiana University.
48.
Luce, P.A., Feustel, T.C., and Pisoni, D.B. (1983). Capacity demands in short-term memory for synthetic and natural speech. Human Factors, 25, 17–32.
49.
Luce, P.A., and Pisoni, D.B. (1983). Capacity-demanding encoding of synthetic speech in serial-ordered recall.Research on Speech Perception Progress Report No. 9 (pp. 295–309). Bloomington, IN: Indiana University.
50.
Manous, L.M., and Pisoni, D.B. (1984). Effects of signal duration on the perception of natural and synthetic speech. Research on Speech Perception Progress Report No. 10 (pp. 311–321). Bloomington, IN: Indiana University.
51.
Manous, L.M., Pisoni, D.B., Dedina, M.J., and Nusbaum, H.C. (1985). Comprehension of natural and synthetic speech using a sentence verification task.Research on Speech Perception Progress Report No. 11 (pp. 33–57). Bloomington, IN: Indiana University.
52.
Marslen-Wilson, W. (ed.) (1989). Lexical Representation and Process.Cambridge, MA: MIT Press.
53.
Marslen-Wilson, W., and Welsh, A. (1978). Processing interactions and lexical access during word recognition in continuous speech. Cognitive Psychology, 18, 1–86.
54.
Martin, C.S., Mullennix, J.W., Pisoni, D.B., and Summers, W.V. (1989). Effects of talker variability on recall of spoken word lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 676–684.
55.
Mcallister, J. (1991). The processing of lexically stressed syllables in read and spontaneous speech. Language and Speech, 34, 1–26.
56.
McGarr, N.S. (1981). The effect of context on the intelligibility of hearing and deaf children's speech. Language and Speech, 24, 255–264.
57.
McHugh, A. (1976). Listener Preference and Comprehension Tests of Stress Algorithms for a Text-to-Phonetic Speech Synthesis Program (Naval Research Laboratory Report 8015).Washington, DC: Naval Research Laboratory.
58.
McKoon, G., and Ratcliff, R. (1980). The comprehension processes and memory structures involved in anaphoric reference. Journal of Verbal Learning and Verbal Behavior, 19, 668–682.
59.
Mehta, G., and Cutler, A. (1988). Detection of target phonemes in spontaneous and read speech. Language and Speech, 31, 135–156.
60.
Miller, G.A., Heise, G.A., and Lighten, W. (1951). The intelligibility of speech as a function of the context of the test materials. Journal of Experimental Psychology, 41, 329–335.
61.
Mirenda, P., and Beukelman, D.R. (1987). A comparison of speech synthesis intelligibility with listeners from three age groups. Augmentative and Alternative Communication, 3, 120–128.
62.
Monaghan, A.I.C. (1992). Heuristic strategies for the higher-level analysis of unrestricted text. In G. Bailly, C. Benoit, and T.R. Sawallis (eds.), Talking Machines: Theories, Models, and Designs (pp. 143–161). Amsterdam: North-Holland.
63.
Monaghan, A.I.C., and Ladd, D.R. (1990). Symbolic output as the basis for evaluating intonation in text-to-speech systems. Speech Communication, 9, 305–314.
64.
Monsen, R.B. (1983). The oral speech intelligibility of hearing-impaired talkers. Journal of Speech and Hearing Disorders, 48, 286–296.
65.
Moody, T., and Joost, M. (1986). Synthesized speech, digitized speech, and recorded speech: A comparison of listener comprehension rates. Proceedings of the Voice Input/Output Society, Alexandria, VA.
66.
Murdock, B.B., Jr. (1962). The serial position effect of free recall. Journal of Experimental Psychology, 64, 482–488.
67.
Myers, J.L., and Duffy, S.A. (1990). Causal inferences and text memory. In A.C. Graesser and G.H. Bower (eds.), The Psychology of Learning and Motivation, Vol. 25: Inferences and Text Comprehension (pp. 159–173). San Diego, CA: Academic Press.
68.
Myers, J.L., Shinjo, M., and Duffy, S.A. (1987). Degree of causal relatedness and memory. Journal of Memory and Language, 26, 453–465.
69.
Nusbaum, H.C., Dedina, J.J., and Pisoni, D.B. (1984). Perceptual confusions of consonants in natural and synthetic CV syllables.Speech Research Laboratory Technical Note 84--02.Bloomington, IN: Indiana University.
70.
Nye, P.W., and Gaitenby, J.H. (1973). Consonant intelligibility in synthetic speech and in a natural speech control (modified rhyme test results).Haskins Laboratories Status Report on Speech Research, SR-33 (pp. 77–91). New Haven, CT: Haskins Laboratories.
71.
Nye, P.W., and Gaitenby, J.H. (1974). The intelligibility of synthetic monosyllabic words in short, syntactically normal sentences.Haskins Laboratories Status Report on Speech Research, SR-37/38 (pp. 169–190). New Haven, CT: Haskins Laboratories.
72.
Nye, P.W., Ingemann, F., and Donald, L. (1975). Synthetic speech comprehension: A comparison of listener performances with and preferences among different speech forms.Haskins Laboratories Status Report on Speech Research, SR-41 (pp. 117–126). New Haven, CT: Haskins Laboratories.
73.
O'brien, E.J., Duffy, S.A., and Myers, J.L. (1986). Anaphoric inference during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12, 346–352.
74.
Pasdeloup, V. (1992). A prosodic model for French text-to-speech synthesis: A. psycholinguistic approach. In G. Bailly, C. Benoit, and T.R. Sawallis (eds.), Talking Machines: Theories, Models, and Designs (pp. 335–348). Amsterdam: North-Holland.
75.
Perfetti, C.A. (1985). Reading Ability.New York: Oxford University Press.
76.
Perfetti, C.A., and Roth, S. (1981). Some of the interactive processes in reading and their role in reading skill. In A.M. Lesgold and C.A. Perfetti (eds.), Interactive Processes in Reading (pp. 269–297). Hillsdale, NJ: Erlbaum.
77.
Pierce, L., and Remington, R. (1984). Comprehension of synthetic and natural speech in the presence of competing human speech. Proceedings of the Voice I/O Systems Applications Conference '84, September 11--13, 1984, Arlington, VA.
78.
Pisoni, D.B. (1981). Speeded classification of natural and synthetic speech in a lexical decision task. Journal of the Acoustical Society of America, 70, S98.
79.
Pisoni, D.B. (1987). Some measures of intelligibility and comprehension. In J. Allen, M.S. Hunni-cutt, and D.H. Klatt (eds.), From Text to Speech: The MITalk System (pp. 151–171). Cambridge, UK: Cambridge University Press.
80.
Pisoni, D.B., and Hunnicutt, S. (1980). Perceptual evaluation of MITalk: The MIT unrestricted text-to-speech system. In 1980 IEEE International Conference on Acoustics. Speech and Signal Processing (pp. 572–575). New York: IEEE.
81.
Pisoni, D.B., Manous, L.M., and Dedina, M.J. (1987). Comprehension of natural and synthetic speech: Effects of predictability on the verification of sentences controlled for intelligibility. Computer Speech and Language, 2, 303–320.
82.
Pollack, I. (1959). Message uncertainty and message reception. Journal of the Acoustical Society of America, 31, 1500–1508.
83.
Pollack, I. (1964). Message probability and message reception. Journal of the Acoustical Society of America, 36, 937–945.
84.
Ralston, J.V., Lively, S.E., Pisoni, D.B., and Rivera, S. (1990). Measuring the workload of comprehending spoken discourse: A first report.Research on Speech Perception Progress Report No. 16 (pp. 155–167). Bloomington, IN: Indiana University.
85.
Ralston, J.V., Pisoni, D.B., Lively, S.E., Greene, B.G., and Mullennix, J.W. (1991). Comprehension of synthetic speech produced by rule: Word monitoring and sentence-by-sentence listening times. Human Factors, 33, 471–491.
86.
Rosson, M.B. (1985). Using synthetic speech for remote access to information. Behavior Research Methods, Instruments, and Computers, 17, 250–252.
87.
Rundus, D. (1971). Analysis of rehearsal processes in free recall. Journal of Experimental Psychology, 89, 63–77.
88.
Salasoo, A., and Pisoni, D.B. (1985). Interaction of knowledge sources in spoken word identification. Journal of Memory and Language, 24, 210–231.
89.
Samuel, A. (1981). Phonemic restoration: Insights from a new methodology. Journal of Experimental Psychology: General, 110, 474–494.
90.
Schneider, W., and Shiffrin, R.M. (1977). Controlled and automatic human information processing: I. Detection, search and attention. Psychological Review, 84, 1–66.
91.
Schwab, E.C., Nusbaum, H.C., and Pisoni, D.B. (1985). Some effects of training on the perception of synthetic speech. Human Factors, 27, 395–408.
92.
Shields, J.L., Mchugh, A., and Martin, J.G. (1974). Reaction time to phoneme targets as a function of rhythmic cues in continuous speech. Journal of Experimental Psychology, 102, 250–255.
93.
Simpson, C.A. (1975). Occupational Experience with a Specific Phraseology: Group Differences in Intelligibility for Synthesized and Human Speech. Paper presented at the ninetieth meeting of the Acoustical Society of America, San Francisco.
94.
Simpson, C.A., and Navarro, T. (1984). Intelligibility of Computer Generated Speech as a Function of Multiple Factors. Paper presented at the National Aerospace and Electronics Conference, Dayton, OH.
95.
Slowiaczek, L.M., and Nusbaum, H.C. (1985). Effects of speech rate and pitch contour on the perception of synthetic speech. Human Factors, 27, 701–712.
96.
Stanovich, K.E. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16, 32–71.
97.
Stanovich, K.E., and West, R.F. (1981). The effect of sentence context on ongoing word recognition: Tests of a two-process theory. Journal of Experimental Psychology: Human Perception and Performance, 7, 658–672.
98.
Terken, J., and Lemeer, G. (1988). Effects of segmental quality and intonation on quality judgments for texts and utterances. Journal of Phonetics, 16, 453–457.
99.
Terken, J., and Nooteboom, S.G. (1987). Opposite effects of accentuation and deaccentuation on verification latencies for Given and New information. Language and Cognitive Processes, 2, 145–163.
100.
Thibadeau, R., Just, M.A., and Carpenter, P.A. (1982). A model of the time course and content of reading. Cognitive Science, 6, 157–203.
101.
Tyler, L.K., and Warren, P. (1987). Local and global structure in spoken language comprehension. Journal of Memory and Language, 26, 638–657.
102.
West, R.F., and Stanovich, K.E. (1978). Automatic contextual facilitation in readers of three ages. Child Development, 49, 717–727.
