Why do we turn images upside down again rather than dealing with them directly, still vertically rotated after passing through our lens?
I don't see how that would cause any problems, and how we'd ever be able to figure out if we are presented with flipped images after getting used to interacting based on visual input, whether flipped or not.
What am I missing?
The basis of this question is a common misconception, and unfortunately the accepted answer by @CHM is also based on this common misconception. The misconception is based on the homunculus falacy: the tendency for people to think that the image that lands on the retina is somehow 'assembled' and presented for something (the 'consciousness') to view. This is not the case.
As the comment by @mgkrebbs explains, there is no orientation (up or down) in the brain, there is only neural firing. The information of the visual scene is distributed over the brain, and information does not have physical properties like orientation. Although as @nico pointed out the neurons that process the information do have a spatial structure that mimics that of the retina, this is a topological property (i.e. stimuli that are close on the retina are processes by neurons that are close in V1) and such a topological property does not induce an orientation.
The root of the problem is really that the question "How do we know the brain flips images projected on the retina back around?" is a pseudo-question. Although it is grammatically well-formed, it makes no semantic sense. When the image is 'in the' (i.e. being processed by the) brain it no longer has physical properties like orientation. Thus you cannot ask if it has been flipped or not.
I think you're oversimplifying the problem. Think about gravitation.
Take this informal example: at any point on a massive sphere, you can define "bottom" such that any massive object with weight less than that of the sphere is subject to an attractive force towards that point. The direction of the force points in the direction of the "bottom".
Assuming this holds true for every massive object, the following is logical: for the purpose of orientation, it is a lot more energy efficient for an organism to simply invert the image using brain circuitry than invert every single sensory input to effectively reverse the definition of "bottom", which follows from physical laws to which the organism is invariably bound.
My answer was initially posted in response to this duplicate question. Since this post was made earlier, the question I answered will likely be closed as a duplicate. I think my answer has something useful to add, so I've added it here, with a little more discussion in response to the other answers:
How do we know the brain flips images projected on the retina back around?
You examine your own visual sense and, possibly, if you're examining patients on a neurology ward, examine theirs as well. Here @CHM's answer is on point, as our sense of gravity (through the vestibular system) is one of the senses integrated into our ability to visually perceive up as up and down as down. It also reflects the confusion expressed by a particular patient with Reversal of Vision Metamorphopsia (see below) when an upside down cup of tea didn't spill out. This doesn't require anyone to invoke a little man watching a video projected somewhere in the occipital lobe. You just ask yourself (and the stroke patient) to describe the world around them. If it's upside down, if people seem to be standing on their heads, you present them with a series of letters with a different orientation and ask them to identify the normal (right side up) letters.
Here is the bulk of my answer to the earlier question, which describes the various inputs the brain integrates in order to produce a visual perception of a right side up (and left-side left) world.
Your retina is presented with an inverted image, but a healthy brain (re)orients the image. I would note that you can talk about the image an individual perceives without invoking a video playing in the brain.
The spatial orientation of a perceived image is a complex process that seems to involve integration of information about the relative position of objects from the light perceived by the retina, about the position of the eyes from the muscles of the eyes, and about the position of the head from the vestibular system and muscles of the neck. It may or may not involve other body position information from the cerebellum and proprioceptive cortex.
As with much of our knowledge about the localization of function in the human brain, the evidence for the importance of these various streams of information comes from cases with lesions at a number of different locations that produce brief disruption of the integration of this information, a syndrome called Reversal of Vision Metamorphopsia.
For one patient:
He saw people walking on their heads, and the floor next to his bed appeared to be over his head
For another:
People were seen walking on their heads, the windows close to the ceiling were within reach of his hands, and a cup stood "upside down" on the shelf but the "tea did not spill out" (all the objects including the shelf were upside down).
Typically this (Reversal of Vision Metamorphopsia) is the perception of a completely inverted image as in C above (left becomes right, up becomes down), but other reversals have been reported (A, D, and E above).
