When some of the land mammals came back to the oceans, they seem to be successful in the competition with fish at least in some environment even though they seem to have a huge disadvantage in the need of breathing above the water. So how did they succeed in the competition with fish?
So how did they succeed in the competition with fish? you are making the false assumption that they had somehow to succeed at some form of competition. What lead you to this false assumption is I think either of the following two mistakes..
- A misunderstanding of the principle of competitive exclusion.
- A false assumption about the existing overlap between marine mammals ecological niche and fishes ecological niches
The principle of competitive exclusion states that two species occupying exactly the same ecological niche cannot stably coexist. One must either become extinct or shifts its ecological niche (process that is known as character displacement).
A consequence of the above is that if two species have ecological niches that partially overlap, they can still stably co-exist.
Ecological niche of marine mammals
There are two independent lineages of mammals that evolve marine life (cetacea and sirenia). I don't know which one you have in mind when asking the question and a sensitive complete answer would depend on the lineage of interest. However, in any case, there is such a diversity of ecological niches in fishes that it is hard to really fully address the question.
Marine mammals occupy different ecological niches. There is no question of whether one has an advantage over the other as the species do not compete for the same ressources.
A typically important reason for why marine mammals and most fishes occupy different ecological niches relate to their size differences. Simply put, mammals to be quite big! For example a orca eat fishes (and other things) that had few predators before. There is therefore not much competition involved.
Just to reiterate... in your comment you say
so the main advantage was the size?. Size was one of the biggest reason why they would be able to occupy previously unoccupied ecological niches. But it is not an advantage toward competition with fishes as you seem to be thinking as there probably little competition.