That's not completely true. In mammals, including humans, the left of an image projects to the right of each of your retinae (this is due to the eye working like a camera obscura), and conversely for the right of an image. So you have 2 eyes, each eye receives mostly identical information but one is on the left and one is on the right of your body. You must somehow recombine this information together in the visual cortex. The way this is done is through the optic chiasma, where roughly half the nerves from the nasal parts (the half closer to the center of your body) of each retina cross. Then connect to a relay called Lateral Geniculate Nucleus and then connect to the brain. Below you can see a drawing from Ramón y Cajal, who first proposed this theory, depicting the process.
As you can see the points on each retina that receive information from the same parts of the visual field (here represented by the arrow) project to identical locations in the brain. So there is not a complete decussation (anatomical name for a crossing of nerves) like for muscles, but only a partial decussation of about half in humans.
Although y Cajal was the first to explain exactly why it makes sense to have partial decussation, this phenomenon had been known for a very long time. In facts Newton (hardly an anatomist) discusses that phenomenon in his Optiks and suggested that it might be related to the amount of overlap between the 2 eyes. So the more overlap, the higher the amount of partial decussation. This rule, later called Newton-Muller-Gudden law by Walls, seems to be approximately correct for mammals. However, as noted by Walls, there is no evidence of partial decussation in vertebrates other than mammals. Meaning that the decussation is complete (like muscles). In his Optiks Newton concludes that (I'm paraphrasing) if his theory is right then animals with no binocular overlap should have no decussation, which if he is well-informed is true for chameleons. His theory was indeed right but he was misinformed, chameleons have complete decussation.
Finally non-vertebrates do not have "contra-lateral" brains. They do not have this strange property of having each hemisphere mostly connected to the opposite side of their body. Therefore they have no decussation at all.
So in short: vertebrates-mammals = partial decussation ; vertebrates-non-mammals = total decussation ; non-vertebrates = no decussation.
Although dated, Walls book remain the most complete reference discussing decussation:
Walls, G. L. (1944). The vertebrate eye and its adaptive radiation.