The color you perceive is ultimately dependent on the relative contributions of the color photoreceptors in the retina, namely the red, green and blue cones. The sensation of color, therefore, is physically determined by the wavelengths in the light source and the absorbance spectra of the cones. Thereafter, the neurophysiological wirings of the retina and the brain take over and these paths are pretty well defined too. However, one cannot say anything about the qualitative perception of someone observing a light source. This is the big difference between sensation (peripheral processes including the retina) and perception (central processes).
To quote the late Bach-y-Rita: 'You see with your brain, not with your eyes.'
Everything we perceive ('see') is a subjective construct devised by our brains. No red dwarf star is necessary to pose the possibility that you perceive my perceived red as green, or vice versa. No experiments can be conducted to prove or disprove a hypothesis like this.
In terms of experiments: we can test stuff like 'up' versus 'down', as with the prisms mentioned in the question. This, because a subject can qualitatively point out where an object is perceived relative to others, and that can be objectively measured. However, we cannot test the subjective quality of color. We can test relative color contributions through color matching experiments (Kalloniatis & Luu, 2012). These experiments will tell you which contributions the red, green and blue cones have in a certain color mixture, but they will not tell the researcher anything about the subjectively perceived color by the test subject in a qualitative way.
Reference
- Kalloniatis & Luu. Color Perception. In: Webvision. The Organization of the Retina and Visual System, Moran Eye Center (2012)