This answer says that breathing water is inefficient, and that may act as a limiter to the size of fish. I've heard this in a couple of other places as well, but cannot recall the sources. However, the Wikipedia page on allometry indicates that metabolic rate and oxygen consumption scale logarithmically with mass, which implies that the ratio of required oxygen to mass will be lower in larger animals than smaller ones. As such, I would think that the size of respiratory organs would shrink relative to the over-all size of the animal as it is scaled up. The answer linked above even says:
Gills grow in proportionate scales to body size, generally for every 100% increase in body size there is a 50-90% increase in gill size.
Which is entirely compatible with the (admittedly naive) application of allometric organ scaling. How, then, does access to oxygen limit fish growth?