This is a complicated case, as it can have multiple causes.
The genes which are coding for the long (L, yellow) and medium (M, green) wavelength photopigments are located head-to-tail on the X chromosome. This is the reason why this colors are affected much more and especially in men (which only have one copy of this chromosome).
What seems to happen mostly is that the photopigment of one type of cone cells (and their neuronal connectivity) is missing, while the density of the cone cells is still the same. This means that the cells are they, but are not functioning. See these publications (when you have problems getting the articles, let me know in the comments, I can help):
You need to have faulty cones or be completely devoid of the cones that help you perceive a particular colour. If you do possess cones to visualize a colour, you aren't technically colour blind because you should still be able to detect the colour slightly (reference).