One needs to be careful making broad generalizations about meningitis. The term simply refers to inflammation of the meninges (the outer layer surrounding the brain and spinal cord). Meningitis can occur due to a number of causes, most notably viral and bacterial infections, but can also also be due to fungi, parasites, toxins, cancer, etc. There is a vast range in severity of meningitis depending on the aetiology.
For example, rabies infection can progress to a fatal meningitis. Rabies is a virus. You therefore cannot say that bacterial meningitis is worse than viral meningitis.
That being said, what you have likely heard is that viral meningitis tends to be less severe than bacterial meningitis. This is true not necessarily because viruses are worse at causing meningitis than bacteria, but because the common causes of viral meningitis are often more easily cleared by the immune system.
Non-polio enteroviruses and herpesviruses are the most common causes of viral meningitis. These are the same groups of viruses that commonly cause mild cold or flu-like signs. Usually, when we have these signs, we don't go to the doctor to be treated with antivirals, because in healthy individuals our bodies are usually good at clearing the virus. The same is true when the virus is affecting the meninges. However, this does not mean that these viral infections cannot be severe in some cases.
The most common bacterial causes of meningitis in people are Neisseria meningitidis, Streptococcus pneumoniae, and Haemophilus influenzae type b. The pathogenesis of disease is broadly speaking similar to that caused by viruses, but the inflammatory response of the immune system tends to be more severe with these bacterial organisms compared to the common viral causes of meningitis. Bacterial meningitis is also typically accompanied by sepsis affecting other organ systems. While antibiotics exist that are effective at treating these bacteria, many antibiotics do not cross the blood-brain barrier, which makes treatment of bacterial meningitis challenging.
Finally, it is worth mentioning that widespread vaccination may have had an effect on our perception of meningitis severity. For example, at one time polio was a major cause of viral meningitis, but vaccination has vastly reduced rates of this disease in many parts of the world. Similarly, mumps and measles used to be more common causes of viral meningitis, however due to widespread use of the MMR vaccine these are now less common causes of viral meningitis. On the bacterial causes, there are now several common vaccines such as meningitis B/C/Hib/pneumococcal vaccines which are effective at reducing rates of infection with these bacterial organisms.