Audio-Visual Simultaneity Judgments

Article excerpt

The relative spatiotemporal correspondence between sensory events affects multisensory integration across a variety of species; integration is maximal when stimuli in different sensory modalities are presented from approximately the same position at about the same time. In the present study, we investigated the influence of spatial and temporal factors on audio-visual simultaneity perception in humans. Participants made unspeeded simultaneous versus successive discrimination responses to pairs of auditory and visual stimuli presented at varying stimulus onset asynchronies from either the same or different spatial positions using either the method of constant stimuli (Experiments 1 and 2) or psychophysical staircases (Experiment 3). The participants in all three experiments were more likely to report the stimuli as being simultaneous when they originated from the same spatial position than when they came from different positions, demonstrating that the apparent perception of multisensory simultaneity is dependent on the relative spatial position from which stimuli are presented.

Historically, some researchers have argued that people do not experience a strong perception of simultaneity for stimuli presented to different sensory modalities (such as audition and vision; see, e.g., Fraisse, 1964; Guinzberg, 1928; Piéron, 1952). For example, Piéron (1952, p. 295) claimed that whereas a "rigorous impression" of simultaneity can occur for stimuli presented within a sensory modality, no clear sensation of simultaneity is perceived for stimuli presented to different sensory modalities. A similar point was made by Fraisse (1964, p. 109) when he stated that "it is very difficult to assess the simultaneity of two sensations which have nothing in common; this is true for stimulations involving the same sense, but even more so for heterogeneous stimulations."

On the other hand, contemporary research provides a number of demonstrations that appear to show that people can judge the apparent simultaneity of events across the senses (see Spence, Shore, & Klein, 2001, for a review). In fact, a growing body of behavioral and neurophysiological research now highlights the crucial role that temporal synchrony and spatial coincidence play in modulating the effects of multisensory integration (e.g., Lewald, Ehrenstein, & Guski, 2001; Slutsky & Recanzone, 2001 ; see Driver & Spence, 2000, and Stein & Meredith, 1993, for reviews). Research also suggests, however, that temporal and spatial factors may play somewhat distinct roles in modulating multisensory integration effects (for reviews, see Calvert, Brammer, & Iversen, 1998; Munhall & Vatikiotis-Bateson, 2004; Spence & Driver, 2004; Stein & Meredith, 1993). This is clearly true for the case of audiovisual speech perception, in which temporal desynchronization of the auditory and visual components has been shown to have a far more detrimental effect on speech perceptibility (e.g., McGrath & Summerfield, 1985; Pandev, Kunov, & Abel, 1986) than does spatial misaligment (e.g., Bertelson, Vroomen, & de Gelder, 1997; Jones & Munhall; 1997; Radeau & Bertelson, 1977).

Researchers have developed several experimental paradigms to assess the perception of multisensory simultaneity in humans, including the temporal order judgment (TOJ) task and the simultaneity judgment task. In a typical simultaneity judgment experiment, pairs of auditory and visual stimuli are presented at a range of different stimulus onset asynchronies (SOAs), using the method of constant stimuli (e.g., Spence, Shore, & Klein, 2001), and participants are required to judge whether the stimuli were presented simultaneously or successively (e.g., Engel & Dougherty, 1971; Exner, 1875; Hirsh & Fraisse, 1964; Slutsky & Recanzone, 2001; Stone et al., 2001). Psychophysical analysis of the results of such studies is used to determine the SOA at which the participants would have been most likely to have made a "simultaneous" response, known as the point of subjective simultaneity (PSS). …