In the back of my mind I have the idea that human eyes can't notice the color of objects in the far periphery, and that any subjective perception of colors is done by the brain that tries to fill in the color percept.
Is that true?
It depends to some degree on what you define as peripheral vision. Until relatively recently, color vision in the peripheral field has been thought to be substantially less developed than color vision in the central field.
Most estimates of peripheral color perception place the limit of trichromatic (RBG) vision at no more than 30 degrees from fixation; the visual field was thought to be completely color blind at about 50 degrees of eccentricity.
Within the last 30 years, an increased understanding of the changing spatial scale in the peripheral field has led researchers to reevaluate what is believed about peripheral function. Virtually every measure of peripheral color perception can be improved by using a suitably large stimulus in the peripheral field.1
Here is a schematic of distribution of cones in the retina. While cones are decreased in number at 60°, they do exist.2
Here is a schematic of angles on peripheral vision.2
Previous measured sensitivity curves farther into the periphery than thought. In one paper, detection thresholds for colored disks showed that color vision, in particular color vision mediated by the L – M cone-opponent channel (luminance or blue–yellow colors), persists up to at least 50 deg. The larger the color disc, the better the peripheral color vision.
The range of eccentricities over which red–green color vision is still possible is larger than previously thought. Color stimuli can be reliably detected and identified by chromatically opponent mechanisms even at 50 deg eccentricity. Earlier studies most probably underestimated this range. Differences could be caused by technical limitations and the use of stimuli of non-optimal size. (Emphasis mine) In agreement with previous studies we found that the decline in reddish-greenish L − M color sensitivity was greater than for luminance and bluish-yellowish S − (L + M) signals. We interpret our findings as being consistent with a functional bias in the wiring of cone inputs to ganglion cells (Buzás et al., 2006) that predicts a decrease but not a lack of cone-opponent responses in the retinal periphery.
There are two types of photoreceptor in the eye, known as rods and cones. Rods are spread equally throughout the whole retina, whereas cones are focussed in one spot known as the fovea:
Cones are the only type of photoreceptor that can sense colour, so rods can only sense black and white. However, rods can work in much lower light and will also work when not being focussed on. Due to what is known as 'summation of action potentials', through which nerve impulses from multiple rods add together to cause a signal to be sent to the brain, rods are much more effective than cones at sending impulses. If you've ever seen something flickering in the corner of your eye and not been able to see what it is when you've looked properly, this is probably why (Wikipedia: Peripheral vision characteristics).
Edit: Further information on how vision is perceived can be found in this answer.
As for the question of whether or not the brain tries to guess the colour I don't know. I would suggest yes, the brain compensates as much as possible for a lack of information. It's known that the brain can compensate for the blind spot where the optic nerve passes out of the eye and compensates for it using the other eye and other information (Why every human has a blind spot, and how to find yours). However that is just my opinion, I haven't been able to find any research relating to the idea.