I think a main problem here is limiting logic.
By eliminating both toxic/poisonous plants, plants that have evolved to discourage you from eating them, and improperly "washed" plants, things which contain non-plant mater that can make you sick, you are not left with the possibility of diseases.
BSE is caused by prions, which are mater of serious debate and research in academics at the moment. The leading cause of transmission is likely ingestion (1, 2), but it is quite possible that it arises as a mutation sporadically in certain mammals (3).
Coming back to your question, let's suppose two scenarios:
First let's suppose that a prion, or really any other pathogen, were to fall onto a piece a fruit from another mammal. If you, or a cow, were to eat it in this unclean state you have the chance to develop disease. But we would not eat the plant in this unclean state because we are to have it properly washed. The same is true for pathogenic microorganisms that are naturally occurring on plants. I think you addressed this well in the question.
But let us propose another situation where a plant evolves a new protein, or a new form of a protein as is the case in the prion. For simplicity let's call this protein "X." X may or may not be toxic to the plant, and cause disease in the plant. If it does, it would likely have selective pressure against the population it was in (even if it was infectious to plants like a prion). But we may not, at least in the beginning be able to tell if the plant has the protein in it, so we go ahead and eat it (and for the sake of argument, we won't cook it, which is bad news for all kinds of pathogens).
If that protein causes sickness in an animal (like humans) we would say that it is toxic, and we would exclude it per your preamble. If it is stable enough, it would remain toxic in our bodies, and should anyone be unfortunate enough to eat us or roll around in are carcass, there would be a small but theoretical chance they could get sick as well. Again it would likely be excluded from this question as a toxin.
I think these get at what you are asking, but I think there is a spirit of what you are asking which goes further:
Why aren't contagious agents produced by plants?
Miguel addresses the problem quite nicely. First, the genetic distance between humans and plants is so vast that a prion-like protein developing is highly unlikely. Second, plants themselves are unlikely to make good pathogens. Finally, natural selection has favored the production of toxins and irritants when plants have sought chemical means to deter animals (as opposed to a physical deterrent like a thorn).
Prions work because of shared/similar proteins existing within species. A similarly shared protein that we would had with plants would probably be so basic that it was shared by the vast majority of eukaryotes, and if it existed, would likely be capable of destroying a good deal of life, including the plant itself. Thankfully most of the very basic proteins we make a well regulated, and have mechanisms in place to “throw away” miss folded questions.
If we move past the genetic distance, and ignore the conditions that would be needed to select for plant being pathogens, let’s look at other obstacles plants would face. Plants aren’t very good at locomotion, so it would need to get on us by direct contact, carried to us from the elements, or by some other animal.
A good case example would be C. immitis, a fungus. It needs the spores to be kicked up and inhaled by an individual who is in some way compromised (normally). Once this happens, it can begin the next hard step which is to grow.
The immune system of a healthy individual would definitely recognize any fungal or plant matter as foreign object and try to get rid of it. Also note that a pathogenic plant would be limited in size by our behavior. I think even a sloth would not allow a flower or tree to grow out of its back or arm. Anything big enough would cause itching and/or illicit a behavioral effort to physically remove it. Many microbial infections do cause itching, which if you are larger than a microbe, is a good way to have your whole body scratched off your host.
But let’s assume that our pathogen has made it on a human, found a place where it can grow, found enough food to live there (presumably us), has matched our immune system, is small enough that we haven’t been able to physically remove it (think algae), and/or made it to an internal part of body like our lungs where we have a harder time getting rid of it. After doing all that, it has to grow to the point where it can get back out of bodies, and then back into another human and start over. A prion-like pathogen would also have to replicate to significant enough numbers to be transmitted host to host. Transmission is a very difficult process, and as Miguel points out, normally requires a good deal of species dependent specialization and adaption
I have hard time imaging how a plant, even a small one like alga, could undergo enough selective pressure to become an infectious agent. It would be further complicated by that fact that you were eating it in your question, which means it would have to contend with the GI track. The genetic distance makes a prion like pathogen highly improbable if not outright impossible. Toxins, however, are in great abundance in the plant world, and seem to be a good way to get animals to leave you alone.
When I think about plant toxins, I always like to think of the Sumac family. Cashews evolved to be delicious and were cultivated for it, poison ivy evolved to be painful and is destroyed for it, and when we think of the family, we always think of the pain first. [Sorry I know that’s entirely off topic, feel free to delete via edit.]
(1) Johnson CJ, Pedersen JA, Chappell RJ, McKenzie D, Aiken JM (2007) Oral Transmissibility of Prion Disease Is Enhanced by Binding to Soil Particles. PLoS Pathog 3(7): e93. doi:10.1371/journal.ppat.0030093
(2) Bartelt-Hunt SL, Bartz JC (2013) Behavior of Prions in the Environment: Implications for Prion Biology. PLoS Pathog 9(2): e1003113. doi:10.1371/journal.ppat.1003113
(3) Liberski PP. (2012) Historical overview of prion diseases: a view from afar. Folia Neuropathol. 50(1):1-12.