Link to the webpage: https://dbechar.github.io/Audio-visual-search-task/
We perform visual searches multiple times a day. Before leaving the house, for example, we might have to search for our keys. But how exactly do we know where to look? Research in the field of visual perception and search has mostly focused on understanding and examining unnatural, abstract tasks and simplistic stimuli such as lines and color patches. This research has helped tremendously in understanding the basic properties of perception, however, we perceive much more than just simple lines. The environment we act within and interact with is much more complex than that and us humans receive input from many different modalities that help us make sense of it, e.g. acoustic and visual input. Thus, research focusing solely on simplified stimuli and the visual input during search does not suffice to explain human perception and behavior in its entirety.
Vision is generally considered to dominate the multisensory perception of the world. Shams and colleagues (2000) however have shown that auditory information can qualitatively alter the perception of a visual stimulus. Therefore, they showed that visual perception can be manipulated by other sensory modalities.
I want to see whether hearing and seeing an item at the same time will improve search compared to seeing the item with the sound of another item.
At each time, N = 8 pictures are presented at random locations within a circle on the display such that there is no overlap between the pictures. N will stay fixed throughout all trials. Participants are told an object and asked whether it is present in the following search display, having to indicate their answer via key-press (F-key for "target present" and the J-key for "target absent"). During the search, a sound will be played that either corresponds to the cue or not.
Participants will exhibit faster reaction times and make less errors in congruent trials (trials where sound and image match the cue) compared to incongruent trials (trials where sound and image do not match the cue) during the visual search task using natural stimuli, indicating that congruent multisensory information enhances visual search efficiency.
Similarily, participants will report that the cued target object is not present when it is in fact not present faster, if the sound played does not match the cued target, as it reinforces the believe that the item really is not present in the search display.Both the audio files used in this experiment as well as the images can be found here. The audio files were all taken from findsounds.com and the images were found through web search. They were all resized to 100x100 pixels and the background was removed.
At the beginning of each trial, a fixation cross is presented for 500ms. Afterwards, the target cue is shown (e.g., "dog") for a duration of 800 ms, immediately followed by another jittered fixatin cross that is shown for a duration of 200 to 400 ms. This minimizes participants being able to predict the exact appearance of the search screen. Lastly, the search display is presented in which the target is either present or not. At the same time, a sound will be played that is either congruent or incongruent with the cued target object. The search display including the sound will be displaied either until a response is given or for maximally 5000 ms. Between two trials a blank screen will be shown for 1500ms.
Participants will have to do 15 practice trials first to get used to the task. Afterwards, the experiment begins and participants can start with the actual experiment.