2, 4, 5, and 6.
6 being that the UV light (from the sun), fluctuations in temperature, humidity, wind etc mean that the virions are decayed relatively rapidly for most virus species.
To address (1): In general a virus, such as SARS-CoV-2 or influenza is about 100 nanometres (0.1 micrometres (μm)) in diameter, whereas a pollen is about 10 μm (sizes in links for the respective viruses) - the virus is 100x smaller, but is largely spread through droplet transmission. Droplets are comprised of typical nasal secretions (i.e. snot) and/or saliva and those of about 5-100 μm in size are relatively dense and fall to the floor rapidly. Less than 5 μm can float for quite some time1 but rapidly dehydrate and lose virion integrity, so can't cause infection.
(2) and (4) Infections produce a lot less virus than a tree does pollen. A typical allergenic tree like a Silver Birch (Betula papyrifera) can produce about 2 million pollen grains from a single catkin - up to about 2 billion grains per tree. Multiply that by the number of trees, and you'll get some idea of the number of pollen grains for that species alone. On the other hand an infected person has about 7 million virions per millilitre (ml) of saliva. However, not all of each ml is turned into droplets when speaking and it turns out that only about 37% of 50 micrometre droplets will contain a single virion, and that this drops to 0.37% for 10 μm droplets (see ref 1). This means that each infected person at their infectious peak is only producing putting a tiny proportion of the virus they contain.
(3) and (4) Sort of - plenty of people can be infected at once, as you will have seen during the waves of infection, but once sick they aren't out there walking around constantly emitting virus into the environment, they are in bed, at home or in a hospital (assuming they are following good public health advice). This also ties into the answer for (1) - the droplets just don't last like a pollen grain can. Pollen's purpose is to travel to find a new flower to fertilize, so trees that are wind pollinated have a selective pressure to produce pollen that can last in the environment and still fertilize another tree. Viruses don't have the same selective pressure because transmission relies on "close" contact of a mobile organism - not a tree that sits in the same place and the closest one might be miles away.
(5) How an infection takes place is multifactorial - you need the conditions to be just right for transmission to take place, and a big enough dose of virions to cause an infection (often for things like Influenza or Adenovirus this is in the 3-10 virions range), you then need it to hit the right tissue in the body and evade the immune system. All pollen has to do is hit a mucosal membrane (nose, mouth, eyes etc) for there to be an immune cell there to activate the immune response.
Long story short infection is much harder to do, produces less virions and has lower likelihood of happening.
1: Stadnytskyi V, Bax CE, Bax A, Anfinrud P. The airborne lifetime of small speech droplets and their potential importance in SARS-CoV-2 transmission. Proc Natl Acad Sci U S A. 2020 Jun 2;117(22):11875-11877. doi: 10.1073/pnas.2006874117. Epub 2020 May 13. PMID: 32404416; PMCID: PMC7275719.