Is brain plasticity such that we can train ourself to see with our ears?

Question

I am finishing writing some code which will parse a photo (eventually video) and use all the RGB information to synthesize an audio representation. I am wondering whether a typical person has sufficient neural plasticity to learn how to listen to this audio to understand an image in a general sense? I am not looking for perfection. If the person has good vision they would do well to spend time listening to such synthesized audio while simultaneously viewing reality to give some training to enhance their interpretative abilities. Once trained they could augment or supplant vision with its sonic equivalent.

How plastic is our audio-visual brain? Is there hope this will work ?

PS. Once working I will update this Q

---

For those wondering about the details : I am traversing the image using a Hilbert Curve which tends to preserve spacial relations amongst pixels to minimize re-training upon change to pixel resolution. This flattens the 2D photo into a 1D line sprinkled across from left to right with points storing respective pixel values (at a 1st approximation I collapse RGB into grayscale 0.21 R + 0.72 G + 0.07 B)

... to create the audio representation I visit each pixel position on this line and introduce an audio frequency oscillator per pixel at a unique frequency such that the beginning pixel at far left is given the lowest frequency in our range (say 200 hertz) on upward until the oscillator frequency at the far right pixel renders the highest frequency (say 10 khz) ... the grayscale value drives the volume of that pixel's oscillator

... further details at: isomorphism between video and audio https://www.youtube.com/watch?v=DuiryHHTrjU

Beauty of this approach is it lends itself to performing this transformation in reverse (a bijection) - from audio to video we can use a Fourier analysis ( FFT ) of audio mapped into pixels - then back again to audio, rinse and repeat ...

Answer 1

Short answer
Yes, we can see with our ears.

Background
Bach-y-Rita famously stated "We see with our brains, not our eyes". Bach-y-Rita worked for decades on sensory substitution. Sensory substitution approaches in general aim to replace for a lost sense by redirecting information normally captured by that sense to another still functional one.

Bach-y-Rita focused on substituting vision by tactile information. His Tactile Vision Substitution System (TVSS) captured camera images and translated them to tactile images projected onto the back of blind subjects. His decades of research eventually culminated in the commercially available BrainPort device (Stronks, 2016).

Vision-to-auditory substitution devices have also been developed, most notably the vOICe by Peter Meijer and the prosthesis-substituting-vision-by-audition (PSVA) device, among others. The vOICe translates images into sounds by re-mapping their x-axis into the time domain, and their y-axis into the frequency domain (Meijer, 1992). This “image-sweepline” technique has subsequently been deployed in the EyeMusic device for use in sensory substitution (Abboud et al., 2014). The PSVA makes use of a pixel-by-pixel acoustic frequency transformation,where vertical position is coded as pitch and horizontal position as binaural intensity and phase differences, and brightness into loudness (Capelle et al., 1998).

Peter Meijer refers to the vOICe as generating 'soundscapes', much like the approach you are describing. I seriously encourage you to contact him, he is a very nice guy and open to exchange.

Anyway, to come to your question - can we see with our ears? Yes we can. Several studies have indicated that sounds can successfully be used to perform visual tasks, including localization (De Volder et al., 1999), pattern recognition (Arno et al.,2001a, 2001b) and depth-perception (Renier and DeVolder, 2010).

Several studies have shown that the visually deprived brain is capable of re-routing auditory sensations into visual equivalents by re-routing this information to the de-afferented visual cortex in the blind (reviewed in, e.g., Poirier et al., 2007). This is referred to as cross-modal plasticity of the brain.

However, the learning curve is quite steep, in that a lot of practice is needed to obtain useful information from visual information coded through sound (Stronks et al, 2015). Further, what is probably the biggest issue in analyzing soundscapes is visual clutter. While the visual system is very well capable of filtering out image clutter and extract the useful information, soundscapes will inevitably turn every irrelevant object in the scene into a corresponding sound. Front-end visual processing techniques are likely to be key to turning an auditory visual substitution device into a practically feasible device for the blind, or the sighted for that matter.

The visual cortex is recruited for auditory tasks in the blind, but not in the sighted. However, I am inclined to believe it is practice, and not visual deprivation per se, that determines the performance with sensory substitution approaches in general (Stronks et al, 2015).

<sub>References
- Abboud et al., Rest Neurol Neurosci (2014); 32: 247–57
- Arno et al., Neuroimage (2001a); 13(4): 632–45
- Arno et al., App Cog Psych (2001b); 15(5): 509–19
- Capelle et al., IEEE Trans Biomed Eng (1998); 45(10): 1279 - 93
- De Volder et al., Brain Res (1999); 826(1): 128–34
- Meijer, IEEE Trans Biomed Eng (1992); 39: 112–21
- Poirier et al., Neurosci Biobehav Rev (2007); 31(7): 1064–70
- Renier and De Volder, J Integr Neurosci (2010); 4(4): 489
- Stronks et al., Brain Res (2015); 1624: 140–52
- Stronks et al., Expert Rev Med Dev (2016); 13(10): 919-31</sub>

Answer 2

I would like to add to the great answer by AliceD with my own experiment. I wish to add this because the results, while intellectually natural, were intuitively fascinating, and because my sample size is tiny (read: N=1), and I'd love to have others go out and repeat the experiment.

I don't quite show that you can see with your ears, but I do think it shows that you can hear with your eyes. (Or, more formally, we fuse our senses in a remarkable way)

The bill of materials is daunting:

• Two spoons
• Willing test subject

In my case, the subject was my girlfriend at the time.

It is known that we use spectral effects to determine the elevation angle of a sound. Our ears and our shoulders are known to shape the sound, emphasizing some frequencies and muting others. Of course, this process is impossible to do if you don't know what the sound was "originally." If you don't know what the sound was as it reached you, you can't figure out how much the ears have colored it.

For this experiment I picked two spoons, and verbally described what I was going to do. I was going to click the spoons in various places and have her reach for them. I intentionally did not click the spoons together in this demonstration because I did not want her to get to hear the spoons until the experiment started.

I first asked her to close her eyes. I clicked the spoons in various places, and she identified which direction she thought it came from. She was reliably accurate in left/right direction, because she could hear the time delays between the sound hitting her left and right ear. However, her elevation angle was rather random. There did not appear to be any pattern at all to it.

Then I asked her to open her eyes, and do a very boring version of this experiment. With her eyes open, she of course was able to point to the clicking spoons 100% of the time. (This was actually the hardest part, because it seems absurd to the test subject. They start to think it's a trick).

Then, I had her close her eyes again, and repeat the experiment. The results? She was dead on every time. Didn't miss a single one, both azimuth and elevation.

So what do I claim happened? When we started the experiment, she did not know the frequency spectra of the spoons clicking. As such, she could not effectively back out what transform her ears and shoulders were applying, and could not figure out elevation angles. She could make some guesses, as a human that has heard things clacking together, but that was insufficient to accomplish the task.

In the boring second phase, she could now fuse the information she got from her eyes in with that from her ears. Now she could determine what angle the sound was coming from with her eyes, figure out what the transform of her ears/shoulders should be, and back out what the "true" sound of the spoons was.

In the third phase, she now knew the "true" sound of the spoons, so whenever soundwaves hit her ears, she could use that knowledge to figure out what transform her ears/shoulders must have been applying, and figure out angles.

I used this to argue that we can hear with our eyes, but it might be more effective to say that the brain doesn't pay as much attention to the division between the 5 senses as we do when we talk about our sensory experience. As far as it is concerned, it's all simply neural stimuli. It will fuse these into one coherent image, and that is what really matters.