I am finishing writing some code which will parse a photo (eventually video) and use all the RGB information to synthesize an audio representation. I am wondering whether a typical person has sufficient neural plasticity to learn how to listen to this audio to understand an image in a general sense? I am not looking for perfection. If the person has good vision they would do well to spend time listening to such synthesized audio while simultaneously viewing reality to give some training to enhance their interpretative abilities. Once trained they could augment or supplant vision with its sonic equivalent.
How plastic is our audio-visual brain? Is there hope this will work ?
PS. Once working I will update this Q
For those wondering about the details : I am traversing the image using a Hilbert Curve which tends to preserve spacial relations amongst pixels to minimize re-training upon change to pixel resolution. This flattens the 2D photo into a 1D line sprinkled across from left to right with points storing respective pixel values (at a 1st approximation I collapse RGB into grayscale 0.21 R + 0.72 G + 0.07 B)
... to create the audio representation I visit each pixel position on this line and introduce an audio frequency oscillator per pixel at a unique frequency such that the beginning pixel at far left is given the lowest frequency in our range (say 200 hertz) on upward until the oscillator frequency at the far right pixel renders the highest frequency (say 10 khz) ... the grayscale value drives the volume of that pixel's oscillator
... further details at: isomorphism between video and audio https://www.youtube.com/watch?v=DuiryHHTrjU
Beauty of this approach is it lends itself to performing this transformation in reverse (a bijection) - from audio to video we can use a Fourier analysis ( FFT ) of audio mapped into pixels - then back again to audio, rinse and repeat ...