Lecture Topics

•    sensory outfit and ecological niches (sensors co-evolve with behaviour, with occupation of specific environments)
•    super-human senses in the animal kingdom: UV, IR, polarised light, electricity, magnetism, …
•    sensory processing strategies and sensory inegration: cortical networks
•    combining sensory infromation in speech: recognising emotions, lip reading, cortical processing of sign language, ventrilocism
•    classical theories of perception: Gestalt psychology, constructivist approach, direct perception
•    perception as information processing: neuroscience
•    Q&A about revision, examination – coursework feedback

comparing the senses

5 senses (proverbial, Aristotelian tradition)

•  perhaps a better physical classification: electromagnetic, mechanical, chemical stimuli
•  are these all channels of sensory input?
•  what about temperature, pain? other senses?

Super(?)-human senses

why don’t we have X-ray vision ? who has ?

the limitations of human sensory systems can best be demonstrated by looking at the ‘sensory ecology’ - in their particular habitats, animals have developed during evolution senses which are unknown to humans

The Ecology of Ultraviolet Vision in Birds & Insects

UV light (wavelength < 400nm) exposure can have immediate and/or long term effects on human eyes.
Photokeratitis : something like a painful sunburn on the front surface of the eyes (snow blindness)
many animals, however, use UV vision (and use protective screening mechanisms in the retina to protect against damage)

>> birds see the world in a very different way to humans !!! 

	peacock feather in colour & UV
the ability to detect UV light affects their behaviour recognition of plumage patterns that is invisible to humans
	this ability is used in communication, in particular mating systems  (Bennett et al. 1996)

in this context, various theories based on co-evolution are developed concerning animal and plant coloration (mimicry, fruit colours etc.)

similar abilities in many insects, which is important for them in the context of food selection and orientation:
for example, this honey bee is homing in on a colt's foot (Tussilago farfara L.) flower (which appears just as brigth yellow to the human eye);
the bee is guided through the prominent UV colour patterns to the centre of the flower head
the flower is usingthis pattern as advertising sign, because it needs the bee to transport its pollen
 

photographed in UV flash (c) Bjørn Rørslett

we find the same story underwater, where UV can help to counteract poor lighting conditions:

	surgeonfish Paracanthurus hepatus appears dark blue, bright yellow and black to us
	dark blue : camouflage against it's background to us, but bright UV to the UV sensitive camera and to any animal with UV vision

The Visual Ecology of Polarized Light

many animals, particularly invertebrates, are sensitive to the plane of polarized light (preferred orientation of electromagnetic waves)
– humans are completely blind to this property !
this provides these animals with an extra dimension of visual information: functional significance poorly understood

example : dungbeetles
living on firm coastal sands in burrows, the dung beetle returns to the burrow along the shortest possible route from extended foraging trips; in this environment visual landmarks are scarce >> they rely on cues in the sky to find way back, using polarisation-sensitive photoreceptors in the dorsal parts of their eyes (see also Dacke et al. 1999)

what do humans do?  

humans use tools, such as sports sunglasses : advanced multi-layer lens technology in polarising sun glasses, eliminating surface glare and reducing unwanted reflections (fishing!)

Infrared Vision

rattlesnakes and other pit vipers are visualising heat !
heat-sensing pits behind each nostril    highly effective in detecting differences in   temperature at large distances (meters)   crucial for hunting of warm-blooded   animals in the dark and as protection   against large  predators!   integrated with visual images in the brain (Newman & Hartline 1982)

and humans?

humas again need tools, for example: Thermal Imaging cameras that are often used to detect insulation, electrical and mechanical problems, as indicated by heat loss ... (here : a thermal image of a feeding dog)

  this technology is used in particular for night vision equipment !!!

Sensing magnetic fields

humans are also using tools to exploit the magnet field of the earth for navigation purposes :
compass (magnetic needle passively orienting parallel to magnetic field)

navigating birds have been demonstrated to use the earth magnetic field for orientation on their extended journeys (thousands of miles)	little is known about the location and function of magnetic sensors & the strategies to exploit such signals

other suspects for the use of magnetic information:

fish (salmon)   turtles   insects (Monarch)

Electric field

some fish can generate electric fields (electrogenic) and/or detect electric fields (electroreceptive)
which are used for active or passive orientation mechanisms: electrolocation

(see Nelson & MacIver 1999)

	Apteronotus albifrons (black ghost knifefish)
electrolocation is also known for the Australian Platypus

The combination of sensory information

why, for what purpose do we need to combine different sensory sources ?

• combined alarm systems: jungle (sounds and spurious visual cues), darkness (sound, smell), braking (vision and sound)
• object identification: information of different modalities support each other
• visual-auditory localisation (twilight, peripheral visual field, occlusion, cluttered scenes)
• reliable information for control of locomotion (e.g. visual-proprioceptive cross-adaptation: Pelah and Barlow 1996)
• retain coherence of the world: object unity, perceiving one’s own presence in space (e.g. cue conflict > motion sickness)

how can cross-modal combination of information be investigated experimentally ? (an example for vision-touch integration: recognise objects by vision and/or haptics)

(large interest from engineering: sensory fusion in robots, VR, video games)

>> how do nervous systems combine sensory information ?

Multisensory neurones

single cell recordings in superior colliculus (subcortical) of the cat input from various other cortical regions (visual, auditory, somatosensory, motor...)
	alignment of receptive fields >> integrated multisensory maps spatial 'coincidence detectors' demonstrate the true combination of cues: only if both stimulus qualities are present at a given location, the neurone would respond - 'crossmodal integration'

after Stein & Meredith 1993

Multisensory regions in the human brain

how and where are the different senses integrated in the human brain ?

fMRI is the tool to study some cases of multisensory integration: most prominent :
combination of visual and auditory information (Calvert et al 1998)

• red: viewing mouth movements without sound
• blue: listening to speech
• yellow: activated by both signals

these could be the regions for crossmodal identification !

Audio-visual speech integration

one step further – conjunction: is coherence between visual AND auditory information crucial?

what happens in the multisensory brain regions when listeners receive conflicting signals (McGurk Effect) ? (seeing someone saying 'ga' while listening to sound 'ba' leads to perception of 'da') (King and Calvert 2001)

speech elicits specific activity in superior temporal gyrus/sulcus (STG & STS)

congruent signals (lip movements synchronised with heard words): larger responses than sum of two components incongruent words (lip movements and audio signals from different words): reduced responses

binding of visual and auditory components of speech in (STS) !!!

Encoding of sign language

•   how the deaf process signed languages in the brain ?
•   specialised structures for decoding linguistic patterns in general?

    example for sign language: butterfly       

what is the cortical location for language error handling ? signed sentences mostly encoded in left hemisphere (like parsing spoken language hearing people) identical brain structures for similar tasks in the left inferior frontal cortex in deaf and hearing people meaningless grammatical hand movements >> greater blood flow in the planum temporale (like spoken equivalent in hearing people)

reading emotions

observers are asked to rate the sadness of a series of (real) faces with different expressions between happy and sad

sadness ratings follow a ‘psychometric curve’: from low to high for happy to sad faces combining the face display with a sad or happy voice shifts the psychometric curve to higher or lower sadness ratings

emotional voices influence the categorization of facial expression !!
(De Gelder & Bertelson 2003)

for some more intersting observations about about facial expressions click here ...

transfer across sensory modalities

ventriloquism: speaking or uttering sounds so that they seem to come from the speaker’s dummy or some other source than the speaker for some nice moves, click here for 'read my lips'

synaesthesia: a mixing of senses causing a person to experience such things as colored hearing, gustatory sights, and auditory smells… (one in every 25,000 people !) (Ramachandran & Hubbard 2003)

sensory substitution: replacing a (lost/missing) sense with some other sense! the vOICe system translates arbitrary video images into sounds - this means that you can see with your ears, whenever you want to an interesting case study (on YouTube)...

Human echolocation : see a California teenager who is blind overcome his disability by experiencing life with all his other senses...

Gestalt psychology

Gestalt theory has its focus on the principles of perceptual organisation: ‘the whole is more than the sum of the parts’
‘laws’ describing perceptual phenomena <<you may want to ask yourself: are these explanations?>>
(Wertheimer, Koffka, Kohler)

praegnanz: good shape (vision, acoustical) similarity proximity good continuation familiarity (illusion from Kanizsa 1976)

constructivist approach

emphasizing top-down processes in perception: the mind tries to make the best sense of ambiguous data
(Neisser, Gregory)

active & constructive process iterative comparison of sensory input with internal (stored) knowledge hypotheses & expectations generate specific errors (illusions) physiological basis ? revival: Bayesian estimation ('Dalmatian' from Marr 1982)

direct perception

emphasizing bottom-up processing, exploiting richness of information content in sensory data
direct use of sensory input for behavioural control without need of high-level representation
(Gibson)

comprehensive capture of information in optic array unambiguous information about spatial layout (flowfield) affordances: meaning in behavioural context resonance: process to extract information (filter tuning?) flowfield sketch: pilot approaching runway, from Gibson 1979

the information processing approach

in these lectures the focus was on a neuroscientific & computational approach to perception, which describes the first information processing steps as basis for cognitive psychology

common themes for all 5 major sensory channels are the following key concepts:

• receptor: transformation from external to neural signals
• filter: encoding of information, tuning to specific properties
• receptive field: physiological basis for localisation and tuning
• representation: cortical processing and mapping
• illusions: processing can misrepresent the physical world
• active sensing: sensory system link with exploratory behaviour

how does this approach relate to other concepts of perception?

summary: integration & conceptual framework

•   humans rely on a variety of senses to organise their behaviour
•   many animals have sensory systems that are not available to humans - adaptation to specific environments and behavioural strategies
•   different types of sensory information is integrated in the human brain (audio-visual binding is important for understanding language)
•   ‘classical’ approaches to understand human perception include Gestalt, constructivist and direct perception
•   the currently dominating view focuses on information processing in biological systems (~cognitive psychology)

some things to remember about revision and exams

•   how do you revise for Sensation and Perception? (work in small groups)
•   reading materials : handouts, textbook, lecture webpages, references >>> start at http://www.pc.rhul.ac.uk/staff/J.Zanker/PS1061/PS1061.htm
•   exams format: 2 hours, answer 2 out of 4 questions
•   discuss with peers!
•   practise sample exam script (webpage)…
•   course intranet forum for FAQ

Reading:

•    Goldstein, EB (2002)  Sensation and Perception (6th ed.), Wadsworth (152.1 GOL)
•    Stein & Meredith (1993) The Merging of the Senses. MIT Press (612.8 STE)
•    Eysenck (2001) Principles of Cognitive Psychology. Psychology Press (153.4 EYS) chapter 2 for conceptual frameworks
•    Calvert, G A, Brammer, M J, Iversen, S D 1998 "Crossmodal identification" Trends in Cognitive Sciences 2, 247-253 (look here)

• Bennett ATD, Cuthill IC, Partridge JC & Maier EJ. (1996) "Ultraviolet vision and mate choice in zebra finches." Nature 380, 433-435
• Dacke M, Nilsson D-E, Warrant EJ, Blest AD, Land MF& O´Carroll DC (1999) "Built-in polarizers form part of a compass organ in spiders" Nature 401, 470-473
• de Gelder B, Bertelson P (2003) "Multisensory integration, perception and ecological validity." TICS 7(10), 460-476.
• Ernst MO, Banks MS (2002) "Humans integrate visual and haptic information in a statistically optimal fashion." Nature 415, 429-433
• Kanizsa G (1976) "Subjective Contours" Scient.Am. 234, 48-52.
• Gibson JJ (1979) The ecological approach to visual perception. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
• King, AJ, Calvert, GA 2001 "Multisensory integration: Perceptual grouping by eye and ear" Current Biology 11, R322-R325
• Land, MF, Nilsson D-E (2002)  Animal Eyes.  Oxford: Oxford University Press
• Marr D (1982) Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. San Francisco: Freeman & Co.
• Moller, P. (1995). Electric Fishes: History and Behavior. London: Chapman & Hall
• Nelson, M.E. and MacIver, M.A. (1999) Prey capture in the weakly electric fish Apteronotus albifrons: sensory acquisition strategies and electrosensory consequences. J. Exp. Biol. 202, 1195-1203
• Newman E, Hartline P (1982) The infrared "vision" of Snakes. Scientific American 213:116-127.
• Nishimura H, Hashikawa K, Doi K, Iwaki T, Watanabe Y, Kusuoka H, Nishimura T, Kubo T (1999) "Sign language 'heard' in the auditory cortex" Nature 397, 116
• Pelah, A, Barlow, H B (1996) "Visual illusion from running" Nature 381, 283
• Petitto LA, Zatorre RJ, Gauna K, Nikelski EJ, Dostie D, Evans AC (2000) "Speech-like cerebral activity in profoundly deaf people processing signed languages: Implications for the neural basis of human language" PNAS 97, 13961–13966
• Ramachandran VS, Hubbard EM (2003) "Hearing colours, tasting shapes" Scientific American, April 2003
• Sekuler R, Sekuler AB, Lau R (1997) "Sound alters visual motion perception" Nature 385, 308

to download pdf-file of lecture handout (colour version) click here

back to course outline
last update 21-11-2008
Johannes M. Zanker