Why would things self-organize in such a manner? I'm a mathematician and I have no baggage in ecology or biology. If the question is too broad maybe someone can point me to formal sources where this type of issues are tackled.
Basically, they don't. Ecosystems are pretty much either de facto, delimited by geographical boundaries, or defined by us. For example, an underground lake would be an ecosystem and it is organized in such a way for the simple reason that there is no communication between it and any other system.
Most ecosystems however, do communicate. For example, we call a desert an ecosystem but what of creatures living in the borders? That can survive both the harsh desert and the milder outskirts. Do they belong to the desert ecosystem? To the surrounding one? To both?
Basically, we divide nature into ecosystems because we can study them better that way. Species that form part of the same ecosystem tend to interact in one way or another and species from completely separate ecosystems tend not to. This is not some kind of rule however, it is simply the way we categorize things.
To take a different example, how would you define a city? Where does a city end and the countryside begin? Ecosystems are similar, there are seldom very clear cut lines separating them, they sort of bleed into one another.
In other words, your question is moot, ecosystems don't really self organize as such. The organization that does exist is between the different species and is an emergent quality of the complex system that is the ecosystem. Thinking of ecosystems, however, as distinct and well defined entities is generally wrong.
The definition of ecosystem here is missing, but if you mean "an inter-dependency of different organisms that all live in the same geographic region" then I think that competition and cooperation are big drivers of the establishment of these roles.
Some animals and plants are so interdependent that they evolve together despite being different species. The hawk moths have terribly long probiscus which are co-evolved with flowers with deep cup shapes.
The flowers and the moths create more exclusive relationships - the flowers are more reliable food sources for the adult moths when other pollinators can't get to them and the flowers are more likely to succeed in pollination with the moth feeding of fewer kinds of flowers. Cooperation of this nature tends to be show up spontaneously in the ecological milieu because of the utility and stability it can bring.
Competition is the other side to this side of ecosystem evolution. Because living things reproduce to the point where nutrition and security are scarce (meaning that there isn't enough for every plant/animal) cooperation becomes an important advantage.
An ecosystem is a community of living things that exist in conjunction with and rely on nonliving things like a rock, a sandy beach, air, or water. Each organism has an ideal environment, and they may not be able to survive outside that environment. Thus you see a pond with fish living it in next to a forest with squirrels. The division and self organization is natural: the fish do not do so well on land and the squirrels have trouble surviving underwater so they stay in their own ecosystems.
If your question is more along the lines of "why is there a desert here next to a mountain with a forest on the other side of the mountain." This organization occurs due to chance mitigated by the physical laws of the universe based on things like planetary geography / climate, tectonics, and astrophysics.