As mentioned above the inbreeding required to select smaller and smaller dogs would be problematic because of the myriad of health problems that would be associated with it (see inbreeding depression).
The question about physiological limitations is interesting. The main reason biological organisms often don't scale is the fact that length, area, and volume don't scale in the same way. E.g. if something is twice as big L to 2L the surface area is L to L^2 and the volume is L to L^3 which means that many systems no longer work or require more energy (see Allometry) . For example in insects the distance for oxygen to diffuse to cells is short enough that simple diffusion is possible, in humans that is no longer the case so we have whole circulatory and respiratory systems to, in effect, help reduce the diffusion distance for oxygen.
I haven't used this kind of analysis in a while so am not very confident applying it here but I would suggest that some issues that could arise due to things scaling with length versus area or volume are:
- required heart rate
- maintaining body temperature - many small mammals deal with this by falling into periods of cold torpor (sort of like hibernation) where metabolic rate is suppressed
- fragility of bones (as Cort Ammon mentions this is actually not a problem: https://jeb.biologists.org/content/207/9/1577)