Since mRNA is single-stranded, and (mostly) floats freely within the cytosol, what stops it from folding onto itself (like DNA) and preventing transcription?
It does fold on to itself. There are secondary structures in RNA and some of these secondary structures also have regulatory functions (for example, riboswitches). Some of these structures can also inhibit translation (by different mechanisms such as the masking of the ribosome binding site or analogous eukaryotic sequences, or stalling of ribosome etc). Other non-coding RNAs like tRNAs and rRNAs fold into specific structures that are essential for their function.
The translation initiation factor eIF4A has a helicase activity that helps in resolving some secondary structures and facilitate translation (despite this, some structures can impede translation).
Also check out ribosome profiling experiments.
Due respects to everything given here so far (protein associations etc) in reply here as being accurate and part of your answer. There is however the fundamental distinction between RNA and DNA yet to examine: that whole "deoxy-" matter right there in the 'D' of DNA. That's the main key to why DNA more readily attains a double helix than does RNA.
There are at least two known physico-chemical reasons starting from the presence or absence of that 2' hydroxyl for this: (1) DNA can more readily inhabit either an A- or B- form, (whereas RNA is more limited to the A-form). So DNA has more entropic options as a helix former (2) RNA's 2'-hydroxyl can become deprotonated and thus hydrolyzed more easily. So less stability in the long run, not as good an information storage form, less likely to form complementary structures. Not unlikely, just less likely. Here's a good reference about this.