PS1061: Sensation and Perception
Term I,      THURSDAY 2-4 pm      (Windsor Auditorium)

Lecture 9:   Multisensory Integration

Lecturer: Szonya Durant, szonya.durant@rhul.ac.uk, (Room W 245)

Lecture topics

• The modular brain   
• Examples of interactions between the senses
• Neural mechanisms for integration
• Functional benefits of integration
• Cross-modal attention

The modular brain

It helps to understand the brain to demarcate certain anatomical areas – such as lobes in the cortex. The brain can be divided into different parts called Brodmann areas based on the different types of neurons classified according to their appearance under a microscope.

Brodmann areas: Sometimes (but not always) these areas coincide with functional areas (eg Brodmann area 44 – Broca’s area - the area often associated with language.)

Ascribing function to areas – the separate senses

Single electrode recordings can detect the optimal stimulus for a neuron to respond to e.g. a neuron that responds to sound but not to vision	The example neuron above is a visual neuron - it only show a significant amount of responses (impulses or spikes) to the visual stimulus, not the auditory.
Neurospychology - People with damage to specific areas of the brain often show specific impairments	The image above illustrates the primary sensory cortical areas (the first cortical areas to receive the sensory information) for each of the five senses as found by neuroimging studies.
Neuroimaging has concentrated on ascribing functions to spatial locations in the human brain	E.g. Prosopagnosia – damage to an area of the brain causes people to be unable to recognise faces. See here for more info on this condition: http://www.faceblind.org/research/

Modularity within the senses

Even within one sense different areas are thought to process different parts of the stimulus E.g. areas ascribed to visual domains: MT : motion perception (but also stereo vison, combining the information from both eyes to form a sense of depth) V8 : colour? Fusiform gyrus : face perception	Linking all the named visual areas can lead to a very confusing wiring diagram between different modules that each perfom different visual functions. Smooth pursuit Vergence
Similarly in hearing, different brain areas have different functions Sound localisation Frequency selectivity	A similarly complicated diagram detailing the auditory functional modules.

Benefits of modularity

• Specialised areas may improve efficiency of processing.   
• Losing one area does not mean losing all information.
• Evolution may have forced intact units to not change, but rather additional ones to be added on.

Drawbacks of studying the senses separately

• How much have the methods available to us dictated the view of a modular brain?   
• Not all areas have a clear function.
• The aim of perception is to be able to act – it would make sense to use all cues to an event or object together.
• We perceive attributes as a unified whole.

Heatmaps and regions of interest

The binding problem

We easily combine speed, colour, object perception etc. to see this bus moving.

How do we combine all the information that is processed in different areas at different speeds?    Do special areas exists? Does this happen before awareness?

Natural compensation for lost senses

• After brain damage the brain can often restore a great deal of function.
• This is not done by growing new neurons, rather other parts of brain take on new function.
• This is a problem with studies that try and associate certain deficits with certain brain areas – soon after damage the brain starts compensating.

E.g. Burton et al. (2002) conducted an neuroimaging study in people blind from birth and found they still use their visual cortex. Interestingly, it becomes activated when they are using braille.

Synaesthesia

• From the Greek for “union of sensations”.
• One type of stimulus is associated with a sensory experience in another modality.
• E.g. between senses: “the name Derek tastes like earwax” “the frog croaks blue” or within a sense: “the letter A is orange” .

Characteristics It is idiosyncratic – different people have different associations. It is consistent – a synaesthete always has the same associations when tested over time. It is a genuine sensation, not just a learnt association – difficult to verify. Estimates vary between 1 in 2000 to 1 in 200 people who have Synaesthesia. Most common forms are colour-grapheme and colour-musical notes. Neural basis for synaesthesia It has been suggested that this may be an example of “cross-talk” between visual areas. Synaesthetes may have extra connections or the connectivity between different brain regions may be stronger. Links with multisensory theories - do they have “overactive” multisensory areas? do they have additional cross-modal connections? space as an organizing principle? See these two websites for further info on Synaesthesia : http://web.mit.edu/synesthesia/www/ http://www.syn.sussex.ac.uk/

Eye movements as an example of multisensory integration

• Tightly coupled systems between visual perception and motor.
• Areas involved in these must be sensitive to both visual input for target selection and feedback on eye movement location and also be able to produce movements and records of the movements.
• We already mentioned areas such as the superior colliculus and LIP (lateral intraparietal cortex) that respond both to visual inputs and produce motor outputs as well as respond to eye movements.

Click here to go to the eye movements lecture webpage

Visually guided movements

Movements such as threading of a needle also need close communication between visual and proprioceptive input and motor output to succeed. Online feedback constantly updates hand position according to visual feedback. Sport is good example of very highly tuned and complex interaction between domains.

Vision and touch

Rubber hand illusion Touch is the somatosensory system. The participant has the sensation that the hand belongs to them. This illusion illustrates the importance for the synchronisation of the two senses to form a combined percept. Botvinick, M., & Cohen, J. (1998)

Vision and audition

• Spatial localisation accuracy Vision is better for localising where something is, because of accurate retinotopic maps.
• Timing accuracy Sound gives a more accurate time for an event, because of less delays in the auditory system.

Ventriloquist effect In this case you are relying on the visual input to localise the sound. Again, timing is important, the lips of the puppet have to be moving at the same time as the sound. Higher level cognitive effects will also play a role – the character we ascribe to a voice, gestural movements etc. Alais & Burr (2004) measured the size of this displacement.

McGurk effect Here the video input is influencing your auditory percept. When you close your eyes the auditory percept changes. Mandatory fusion You don’t seem to be able to ignore the visual information - you can't avoid fusing the two senses. It seems in some cases we combine percepts regardless of information required.

Watch the effect on youtube here, try it with your eyes open first and then closed : http://www.youtube.com/watch?v=aFPtc8BVdJk

Stream-bounce effect In this case it is the sound influnces whether you see the balls bounce off ech other or stream past each other. The visual signal is ambiguous and the sound helps diambiguate it. Go to http://www.michaelbach.de/ot/mot_bounce/index.html for a demo - test it on your friends!

Shimojo & Shams (2001) measured the effect of various cues around the time the balls inrecept each other. Note that it works best if the sound occurs a little earlier then the intercept point.

Vision and taste

	Different coloured food - Our perception of taste is strongly influenced by appearance. Levitan et al (2008) looked how different colours made people believe sweets tasted different.
Zampini & Spence (2003) :"THE ROLE OF AUDITORY CUES IN MODULATING THE PERCEIVED CRISPNESS AND STALENESS OF POTATO CHIPS"

Vision and balance (vestibular system)

Taken from Stein & Meredith 1993

These effects are important for astronauts, where visual cues no longer match vestibular cues. The "tumbling room" can be used to study these interactions. The whole room can be moved, mis-matching visual and vestibular cues

Levels of integration

Scientists are investigating which of these connections exists and the role they play in combining sensory modalities. e.g Sagiv & Ward (2006)

Underlying mechanisms

Single cell studies Many cells respond to more than one sensory modality. These cells have connections to them from several different areas of the brain. A multisensory neuron responds to auditory and visual input
‘Superadditive’ neuronal response The response from the neuron to both inputs is greater than the combined response to each input alone
‘Subadditive’ neuronal response The response from the neuron to both inputs is less than the combined response to each input alone, this is observed less often.	Images from Stein & Meredith 1993

• It is typically the case for multisensory integration in individual neurons that unisensory stimuli presented in combination produce an effect different than simply summing the effect of the unisensory stimuli presented separately.
• This shows that there is some kind of combination of the two inputs happening within the neuron.

Rules of integration

• The spatial rule Enhanced multisensory integration if the stimulation arises from the same location in external space – decreased activity or response inhibition if stimuli are in different locations
• The temporal rule Different sensory signals can arrive at the neuron within a broad time window for integration (+/-150ms). Multisensory integration will occur only when stimuli arrive within this ‘window’. Different sensory signals can arrive at the neuron within a broad time window for integration (+/-150ms). Multisensory integration will occur only when stimuli arrive within this ‘window’.
• Inverse effectiveness The weaker the stimuli are individually in causing a response, the bigger the increase in response when they are presented together. .

Behavioural advantages of multisensory integration

Performance in detecting the location of a cue was enhanced with crossmodal stimulation - when both types of stimuli were presented. Stein, Meredith, Honeycut & McDade (1989) as cited in Stein & Meredith, 1993, where image is taken from.
Reaction times are improved for multisensory stimuli (Todd, 1912).
Remapping of spatial coordinates between the senses Humans have sevral different sensory maps: Retinal maps - Visual coordinates are often based on the layout of the retina. Somatosensory maps - arranged according to the surface on the skin and the number of receptors. Motor maps - typically are coded relative to the body position. Auditory maps	We now need to know which point in which map corresponds to each other. It is thought that pre-motor areas may translate sensory maps into motor maps.

Feedback – top down effects of attention

It is not only the multisensory areas we mentioned that are affected. Traditional “unisensory” areas can be affected by multisensory inputs. This may be caused by “feedback” see Maculoso & Driver (2005)

Interaction of attention across the senses

• It is thought that attention is the way that senses are combined. Many cross-modal effects of attention exist (see references).
• Does paying attention to a sound in a certain area help with detecting light in the same area?
• To what extent is attention specific to modalities?

Reading:

•    Zanker, JM (2010)  Sensation Perception and Action , Palgrave (Chapter 12)
•    Goldstein, EB (2007)  Sensation and Perception (7th ed.), Wadsworth (152.1 GOL) (End of Chapter 16)
•    Stein, B.E. and Meredith, M.A. (1993). The merging of the senses. MIT Press.2nd Ed.; click here (download Chapter 1 from virtual resources)

Specific References:

• Alais, D., Burr, D. (2004) TheVentriloquist Effect Results from Near-Optimal Bimodal Integration.Current Biology 14: 257-262
• Beauchamp, M. S. (2005) See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Current Opinion in Neurobiology 15(2):145-153
• Botvinick, M., Cohen, J. (1998) Rubber hands touch that eyes see. Nature 391: 756
• Burton, H., Snyder, A. Z., Conturo, E., Akbudak, E., Ollinger, J. M., Raichle, M. E. (2002) Adaptive Changes in Early and Late Blind: A fMRI Study of Braille Reading. J.Neurophysiol. 87: 589-607
• Driver, J. & Spence, C. (1998) Attention and the crossmodal construction of space. Trends in Cognitive Sciences 2(7):254-261
• Levitan, C. A., Zampini, M., Li, R., Spence, C. (2008) Assessing the Role of Color Cues and People's Beliefs About Color–Flavor Associations on the Discrimination of the Flavor of Sugar-Coated Chocolates. Chemical Senses 33(5):415 -423
• Macaluso, E. & Driver, J. (2005) Multisensory spatial interactions: a window onto functional integration in the human brain. Trends in Neuroscience 28(5) 264-271;
• Sagiv N., Ward J. (2006) Cross-modal interactions: Lessons from synesthesia. Progress in brain research 155, 263-275 ; click here (download from virtual resources)
• Shimojo S., Shams L. (2001) Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Neurobiology 11(4), 505-509
• Simner J., Mulvenna C., Sagiv N., Tsakanikos E., Witherby S. A., Fraser C., Scott K., Ward J (2006) Synaesthesia: The prevalence of atypical cross-modal experiences. Perception 35, 1024-1033
• Spence, C. (2010) The multisensory perception of flavour. The Psychologist 23(9):720-723 ; click here (open access web link)
• Stein, B. E., Stanford, T. R., Wallace, M. T., Vaughan, J. W., Jiang. W. (2004) Crossmodal interactions in subcortical and cortical circuits. in Crossmodal Space and Crossmodal Attention Eds Spence, C. & Driver, J. Oxford University Press ; click here (download from virtual resources)
• Zampini, M., Spence C. (2004) The role of auditory cues in modulating the perceived crispness and staleness of potato chips. J. Sensory Studies 19(5): 347-363

to download pdf-file of e-handout click here

Szonya Durant is not responsible for the content of external Internet sites

back to course outline
last update 12-11-2010
Szonya Durant