Short Answer
Synesthesia happens at a point during processing where we are not dealing with "raw visual input" or "raw auditory input" anymore, but already with more abstract constructs such as "colors" or "sounds".
Full Answer
Your question seems to consider sensory perception as a matter of "inputs" and the brain detecting those inputs, such that for example the only difference, from the brain's point of view, between visual inputs and auditory inputs is that one comes from the ear and the other from the eye.
A more accurate way of seeing it is that our sensory perceptions are really the brain constructing a coherent reality from the information it got from our senses. And I do mean "information" in the most abstract, computer-sciency sense possible. It's not like our eye is a photographic plate that contains "what we see", and the brain just has to reconstruct that. The processing of visual information, by which I mean picking out various features from the light received and deriving meaning from them, starts right in the retina. Some examples from the "14.5 Visual Processing in the Retina" section in this page:
http://nba.uth.tmc.edu/neuroscience/s2/chapter14.html
On bipolar cells:
The two bipolar cell types have different functional properties.
On horizontal cells:
The surround effect, produced by the horizontal cells, enhances brightness contrasts to produce sharper images, to make an object appear brighter or darker depending on the background and to maintain these contrasts under different illumination levels.
On the retinal ganglion cells:
The retinal ganglion cells provide information important for detecting the shape and movement of objects.
Type P retinal ganglion cells are color-sensitive object detectors.
Type M retinal ganglion cells are color-insensitive motion detectors.
On amacrine cells:
There are 20 or more types of amacrine cells based on their morphology and neurochemistry. The roles of three types have been identified. One type
is responsible for producing the movement sensitive (rapidly adapting) response of the Type M ganglion cells.
enhances the center-surround effect in ganglion cell receptive fields.
connects rod bipolar cells to cone bipolar cells, thus allowing ganglion cells to respond to the entire range of light levels, from scotopic to photopic.
Similarly in our ears, the cilia cells that vibrate in response to sound are frequency-dependent, and it is information about those frequencies that gets transmitted to the rest of the brain for processing. In other words the very organ that detects sound also performs a kind of Fourier transform on the sound waves it detects before anything else.
See for example:
https://psychology.stackexchange.com/questions/15274/how-does-the-inner-ear-encode-sound-intensity
This parsing, combining and interpreting of information continues throughout the perceptual circuits of the brain, and not in a straightforward way either - it is hypothesized that there are two independent or semi-independent processing streams for visual information, the ventral and dorsal streams, one which is for identifying objects and the other for guiding actions:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4678292/
There is also information flowing the other way, where more "advanced" brain regions draw conclusions about what's actually there and feed those conclusions back into the more "basic perception" parts to ensure that that's what we see. I.e. perception has both "feedforward" and "feedback" processes, or "bottom-up" and "top-down".
All of this is to say, there is absolutely no way for the brain to confuse information from the retina and information from the ear; the early processing in both of those is too tightly linked to the organ itself, and perception as a whole is inextricably linked to cognition. It's not like a mouse and keyboard that you can plug into different sockets; the retina technically is part of the brain in the first place!
This is not to say visual and auditory senses cannot be confused, obviously synesthesia happens, but it likely happens at a point in the processing where we're not dealing with "raw visual input" or "raw auditory input" anymore, but already more abstract constructs such as "colors" or "sounds".
For example this paper (gotten from the Wikipedia page https://en.wikipedia.org/wiki/Neural_basis_of_synesthesia) :
https://web.archive.org/web/20060527085838/http://psy.ucsd.edu/~edhubbard/papers/JCS.pdf
Hypothesizes that the synaesthetic association of letters with colors is caused by cross-wiring between the area of the visual cortex that processes letters and the one that processes colors.
This paper from the same source:
https://academic.oup.com/brain/article-abstract/118/3/661/321747?redirectedFrom=fulltext
Finds that color-word synaesthetes had, when told words the associated with colors, activation in the language and "advanced" visual processing areas but not in more basic visual processing areas.
Similarly for autism, given that sensory perception happens at every level in the brain, including conscious thought, those perceptions can be blocked or filtered (or not) by the brain at any stage. No need to assume it is at the "raw input" stage.
