Very nice question! You are right, in that small pupils will help to reduce the blurriness of vision caused by refractive errors.
Removal of the lens (aphakia) results in farsightedness. It is caused by a focus point that falls behind the retina as the absence of the lens prevents focusing of the image, causing a blurred image. The amount of blur is proportional to the pupil size (Colichia et al., 2008). See their figure 2 below:.
In the upper part you see a hyperopic eye, which has a lens that cannot accommodate sufficiently, comparable to an aphakic eye. the bottom part shows the result of when a pinhole is applied, namely a reduction of the blurred circle and improvement of the focus. This could also be achieved by having very small pupils such as when ambient light is very bright (and a characteristic of the Moken peoples when diving (Gislén et al., 2003). And indeed, under conditions of low lighting the pupil size will increase and hence blurring will be worse.