Seeing What They Read and Hearing What They Say: Readers' Representation of the Story Characters' World
Klin, Celia M., Drumm, April M., Psychonomic Bulletin & Review
Do readers "see" the words that story characters read and "hear" the words that they hear? Just as priming effects are reduced when stimuli are presented cross-modally on two different occasions, we found reduced transfer effects when story characters were described as experiencing stimuli cross-modally. In Experiment 1, a repeated phrase was described as being part of a spoken message in both Story A and Story B, and transfer effects were found. In Experiment 2, in contrast, when the phrase was described as a written note in one story and a spoken message in the other, reading-time results indicated that readers did not retrieve the meaning of the repeated phrase. The results are consistent with findings indicating that visual imagery simulates visual processing and that auditory imagery simulates auditory processing. We conclude that readers mentally simulate the perceptual details involved in story characters' linguistic exchanges.
When we are immersed in a novel, to what extent is our experience of the protagonist's world similar to our experience of our own world? Do we see, in our mind's eye, the words that she reads in the morning newspaper? Do we hear, in our mind's ear, the morning greeting that she gives to her daughter? If a protagonist leaves a note for her daughter saying "good morning" rather than speaking the words, is the phrase encoded differently by the reader? More specifically, given that people encode some of the perceptual details of the words that they read and hear, rather than simply extracting their meaning, we ask whether people also encode some of the perceptual details of the words that story characters read and hear.
Research clearly indicates that when a word is repeated in two different modalities on two different occasions, there will be less facilitation in processing the second encounter of the word than there will be if the word was presented in the same modality on both encounters (e.g., Jacoby & Dallas, 1981; Westerman, Lloyd, & Miller, 2002). For example, using a variety of implicit-memory tests, Rajaram and Roediger (1993) examined priming effects for visually presented test words. They found reduced priming effects when the initial presentation of the words was auditory, as compared with when it was visual. Although there are a variety of explanations for exactly why this might be the case (see, e.g., Schacter, 1994), these reduced effects indicate that people encode some perceptual details of the words they experience, such as the orthographic or phonological features.
But what happens if a story character is described as experiencing a word in two different modalities? Would a mismatch in the perceptual details of the word, as experienced by the character, reduce the probability that the reader would retrieve the word from memory? To answer this question, we examined repetition effects when a phrase is repeated across two unrelated passages. Previous research on repetition effects (Klin, Drumm, & Ralano, 2009; Klin, Ralano, & Weingartner, 2007) has indicated that comprehension of a phrase in Story B is influenced by the meaning of the phrase in Story A. In these experiments, Story A and Story B had different sets of characters, but both described one character, the communicator, leaving the same written note for a second character, the addressee (e.g., "He sure is great company"). In both Story A and Story B, therefore, the addressee experienced the note visually. Under these conditions, the phrase was reactivated from memory when readers encountered it in Story B.
But what if in one of the two passages the phrase is described as having been spoken by the communicator rather than written? For example, instead of the communicator writing a note, he calls the addressee on the telephone. Although it seems likely that readers will infer that the addressee has processed the phonological details of the words in one case and the orthographic details in the other, we ask whether the reader's memory representation of the phrases will include those perceptual details as well. …