Towards a better understanding of sensory substitution: the theory and practice of developing visual-to-auditory sensory substitution devices

Visual impairment is a global and potentially devastating affliction. Sensory substitution devices have the potential to lessen the impact of blindness by presenting vision via another modality. The chief motivation behind each of the chapters that follow is the production of more useful sensory substitution devices. The first empirical chapter (chapter two) demonstrates the use of interactive genetic algorithms to determine an optimal set of parameters for a sensory substitution device based on an auditory encoding of vision (“the vOICe”). In doing so, it introduces the first version of a novel sensory substitution device which is configurable at run-time. It also presents data from three interactive genetic algorithm based experiments that use this new sensory substitution device. Chapter three radically expands on this theme by introducing a general purpose, modular framework for developing visual-to-auditory sensory substitution devices (“Polyglot”). This framework is the fuller realisation of the Polyglot device introduced in the first chapter and is based on the principle of End-User Development (EUD). In chapter four, a novel method of evaluating sensory substitution devices using eye-tracking is introduced. The data shows both that the copresentation of visual stimuli assists localisation and that gaze predicted an auditory target location more reliably than the behavioural responses. Chapter five explores the relationship between sensory substitution devices and other tools that are used to acquire real-time sensory information (“sensory tools”). This taxonomy unites a range of technology from telescopes and cochlear implants to attempts to create a magnetic sense that can guide further research. Finally, in chapter six, the possibility of representing colour through sound is explored. The existence of a crossmodal correspondence between (equi-luminant) hue and pitch is documented that may reflect a relationship between pitch and the geometry of visible colour space.