If you have ever used a natural bath sponge, you know that the structure of a sponge's spongin skeleton is great for absorbing water and squeezing that water back out. The sponge's skeleton contains millions of tiny holes, all connected in a massive network of internal tunnels, through which the sponge takes in food and eliminates waste. It feeds by use of cells that wave tiny cilia to push water into the sponge. Nutrients are absorbed by the animal and spent water and waste are expelled by the help of similar cells, spinning their flagella to force the water out of the sponge. A third class of cells, comprising a basic immune system, move around the organism engulfing microbes potentially dangerous to the animal.
This unique system is facilitated by the structure of the spongin skeleton. The skeleton needs to be strong enough to retain this complicated microstructure, while also having enough give and flexibility that its network of delicate tunnels don't buckle under the force of water pressure, deep currents, or fish and other matter bumping against it. Notice that, when you squeeze the water out of a sponge, it snaps back into place, the tunnels still open for business. Given that these tunnels are how the sponge eats, this is very important. To build a complex tunnel system with millions of tiny little tubes running through it out of something hard and brittle would make it very easy for tunnels to collapse, cracks to form, and other damage to be sustained. Some sponges, as you mention, have hard, brittle "spicules" in addition to the spongin, but these sponges tend to live in deeper regions and need extra strength to deal with the pressure.