Your retina contains both rods and cones. Cones are color sensitive, slow, and concentrated near the center of your field of vision. Rods are "light" sensitive, fast, and concentrated near the periphery. You want to be able to respond quickly to a threat "in the corner of your eye" without needing to see the color of the threat.
This is nicely explained by this diagram (source: http://hyperphysics.phy-astr.gsu.edu/hbase/vision/imgvis/retina.gif)
So when you see a CRT in the corner of your eye, your rods respond quickly enough that you can tell the light of the CRT is flickering. As you turn your gaze towards it, you get the higher resolution, color rendition of the cones - and lose the temporal resolution.