Hamilton-Fletcher, Giles, Wright, Thomas D and Ward, Jamie (2015) Cross-modal correspondences enhance performance on a colour-to-sound sensory substitution device. Multisensory Research, 29 (4-5). pp. 337-363. ISSN 2213-4794

PDF - Published Version Restricted to SRO admin only Download (1MB)

Abstract

isual sensory substitution devices (SSDs) can represent visual characteristics through distinct pat-
terns of sound, allowing a visually impaired user access to visual information. Previous SSDs have
avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprin-
cipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because
it combines tactile scanning with image sonification) and a new algorithm for converting colour to
sound that is based on established cross-modal correspondences (intuitive mappings between dif-
ferent sensory dimensions). To test the utility of correspondences, we examined the colour–sound
associative memory and object recognition abilities of sighted users who had their device either
coded in line with or opposite to sound–colour correspondences. Improved colour memory and re-
duced colour-errors were made by users who had the correspondence-based mappings. Interestingly,
the colour–sound mappings that provided the highest improvements during the associative memory
task also saw the greatest gains for recognising realistic objects that also featured these colours, in-
dicating a transfer of abilities from memory to recognition. These users were also marginally better
at matching sounds to images varying in luminance, even though luminance was coded identically
across the different versions of the device. These findings are discussed with relevance for both colour
and correspondences for sensory substitution use.

Item Type:	Article
Keywords:	Sensory substitution, vision, colour, sound, touch
Schools and Departments:	School of Psychology > Psychology
Subjects:	B Philosophy. Psychology. Religion > BF Psychology
Depositing User:	Jamie Ward
Date Deposited:	04 Jul 2016 10:56
Last Modified:	07 Mar 2017 05:46
URI:	http://srodev.sussex.ac.uk/id/eprint/61862

View download statistics for this item

📧 Request an update