It depends on the mold. For example, humans can smell octenol in ppb concentrations, presumably long before any toxic effects of this or other compounds produced by molds. Octenol is produced by many molds, always through breakdown of linoleic acid (part of many triglycerides, i.e., fatty oils present in most food).
Update: another example of a non-toxic mold is that causing cork taint. The responsible substances (trichloroanisoles, TCA) can be tasted by humans in wine at 2-4ng (nanograms) per liter wine.
 M. J. Saxby: Food Taints and Off-Flavours., 2. ed., Springer 1995, ISBN 978-0-7514-0263-6, S. 210–212.