Academic journal article Perception and Psychophysics

Use of Visual Information in Speech Perception: Evidence for a Visual Rate Effect Both with and without a McGurk Effect

Academic journal article Perception and Psychophysics

Use of Visual Information in Speech Perception: Evidence for a Visual Rate Effect Both with and without a McGurk Effect

Article excerpt

The McGurk effect, where an incongruent visual syllable influences identification of an auditory syllable, does not always occur, suggesting that perceivers sometimes fail to use relevant visual phonetic information. We tested whether another visual phonetic effect, which involves the influence of visual speaking rate on perceived voicing (Green & Miller, 1985), would occur in instances when the McGurk effect does not. In Experiment 1, we established this visual rate effect using auditory and visual stimuli matching in place of articulation, finding a shift in the voicing boundary along an auditory voice-onset-time continuum with fast versus slow visual speech tokens. In Experiment 2, we used auditory and visual stimuli differing in place of articulation and found a shift in the voicing boundary due to visual rate when the McGurk effect occurred and, more critically, when it did not. The latter finding indicates that phonetically relevant visual information is used in speech perception even when the McGurk effect does not occur, suggesting that the incidence of the McGurk effect underestimates the extent of audiovisual integration.

A well-established finding is that the perception of speech can be highly influenced by vision. The evidence for visual contributions to speech perception has largely come from two lines of research. The first line involves studies in which auditory speech stimuli are presented in noise, either alone or with a simultaneous visual presentation of the talker's articulating face; the intelligibility of the auditory stimuli has repeatedly been found to be greatly improved when the talker's articulating face is visible (e.g., Erber, 1969; Sumby & Pollack, 1954). The second line of research involves studies in which a clear auditory syllable (typically, with an initial stop consonant) is presented simultaneously with a visual presentation of a face articulating a different syllable (with a different, visually contrastive initial consonant). As first demonstrated by McGurk and MacDonald (1976), this often results in a striking phenomenon called the McGurk effect in which the resulting percept is a consonant other than the one presented auditorily (for example, an auditory /ba/ presented with a visual /ga/ is often perceived as /da/). Because the presentation of audiovisual stimuli with a cross-modal discrepancy results in a single speech percept incorporating phonetic information from both modalities, the McGurk effect is generally recognized as strong evidence for audiovisual integration in speech perception (e.g., Fowler, 1986; Green, 1998; Massaro, 1987, 1998; McGurk & MacDonald, 1976; Summerfield, 1987). The McGurk effect is phenomenally very compelling and has been replicated many times with different stimuli under a variety of manipulations (e.g., Green & Gerdeman, 1995; Green, Kuhl, Meltzoff, & Stevens, 1991; Jordan & Bevan, 1997; MacDonald & McGurk, 1978; Manuel, Repp, Liberman, & Studdert-Kennedy, 1983; Massaro & Cohen, 1996; Rosenblum & Saldana, 1996). However, the effect does not always occur. That is, when presented with an audiovisually discrepant stimulus (e.g., an auditory /ba/ with a visual /ga/), subjects will sometimes make an apparent "auditory" response (e.g., /ba/); that is, they identify the consonant in a manner consistent with the information in the auditory signal. To some extent, this depends on the specific audiovisual pairing, since certain audiovisual configurations give rise to a stronger McGurk effect than others. For example, a visual /ga/ tends to produce a fairly robust McGurk effect with an auditory /ba/, whereas a visual /ga/ almost never affects identification of an auditory /da/, presumably because /ga/ is visually similar to /da/ and thus does not produce a sufficient audiovisual discrepancy. Generally, however, it is not uncommon to find that for an audiovisual stimulus that does generate a McGurk effect, repeated presentations of the stimulus over the course of an experiment will result in a McGurk effect on only a percentage of trials (e. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.