If I understand your question correctly, I've seen this idea in many papers, sometimes stated clearly and sometimes in more implicit terms. After a quick look I found a paper which should be relevant as a starting point for you: Mayor et al. 2007. Spectrum of selection: new approaches to detecting the scale-dependent response to habitat. Ecology 88(7).
In my mind, there are two questions here. First, to what extent will low population size in sub-optimal environments (leading to a smaller sample to work with) lead to more uncertain inferences of preferred habitat/substrate; basically saying that noise might swamp signal in sub-optimal environments. Second, to detect selection for a particular habitat/substrate, you must sample a wide enough range of environments so that you capture both preferred and unpreferred habitats and transitions between them. In the paper I linked to this issue is touched upon in the discussion:
Second, evidence of habitat selection depends on the scale of analysis. To understand scale dependence, investigations must encompass a spatial scope sufficiently wide to uncover scale domains of selection and the thresholds or transitions between them. In both cases, spatial scale is the empowering principle, and scaling continua are the means of application.
I've seen similar ideas as this show up in papers on habitat choice in birds living in agricultural landscapes as well, where some species are often found in rather poor environments (because these environments are common). Looking at habitat preferences only in these environments might then give a misleading view, since individual choices are constrained. Also relevant, even if not mentioned in your question, is the concept of maladaptive habitat selection (sometimes called ecological traps), where you can have situations where species use environmental cues that used to indicate high fitness habitats, but might now result in maladaptive choices, see e.g. Pärt el al. (2007). However, in this case you will still have a clear preference/selection (as in your high variability environment), the problem being that the organism is selecting suboptimal habitats from a fitness perspective.