The mycobacterium of TB doesn't secrete any toxins. The cause of disease in the immunocompetent is the collateral damage due to the immune response against disseminated infection. But then, when HIV patients are immunocompromised, why should TB do any harm to them? There is an absence of the typical granuloma in HIV cases.
This is an interesting an useful question about host-pathogen interactions. For a general reference (for both hosts with and without a competent CD4+ T-cell response) see Cecil Medicine, Ch.332, and Murray Medical Microbiology, Ch. 28. There is also a decent review in Lancet Infectious Disease that discusses the pathophysiology in immunocompromised hosts.
Overview / tl;dr
Tuberculous mycobacteria may not secrete any toxins or replicate quickly, but they are clever and persistent. Without an effective immune response, they will eventually overwhelm the lungs, spread to other organs, and kill the host through organ failure and/or embolization. In these cases, infection is universally fatal if not treated. This is what happens in HIV patients with a low CD4+ T-cell count.
In hosts with a competent CD4+ T-cell response, infection is sometimes fully resolved and sometimes held at bay for many years without major organ damage. In other cases, the immune system's methods are too powerful for the infected organ. Tissue damage in these cases is the direct result of prolonged activation of large fused groups of macrophages. Importantly, this is not an overactive immune response, it just happens to be too much for the local organ. You cannot treat TB by shutting down this immune response. If you block TNF-$\alpha$, for example (a major player in the cross talk between CD4+ T cells and macrophages), you end up with disseminated TB. The solution here is elimination of the pathogen with direct anti-mycobacterial treatment.
In either an immunocompetent or immunocompromised host, tuberculous mycobacteria enter, e.g., alveolar macrophages through phagocytosis and, initially, prevent fusion of the phagosome with lysosomes, while allowing fusion with other vesicles. This allows the bacilli to live, access nutrients, and replicate within the macrophage.
In the immunocompetent host, the interaction between the infected macrophage and CD4+ helper T cells leads to formation of an activated macrophage and either elimination of the mycobacteria or compartmentalization of the infection. Either of these outcomes are generally good for the host. Clearance of a pathogenic bacteria is obviously good. Latency in a walled off compartment is a close second.
The typical disease syndrome of pulmonary tuberculosis involves a brief, often unremarkable respiratory infection, followed by a long period of clinical latency. In many cases where the primary infection isn't fully cleared, latency may be lifelong, only to be discovered incidentally at autopsy. The brief illness followed by clinical latency corresponds to the infection of alveolar macrophages with an immune response that successfully walls off the infection.
In a host with generally capable CD4+ T cell / macrophage cross talk, if disease develops, it typically occurs after years or even decades of latency, through reactivation. Reactivation may be the result of a temporary failure of the compartmentalization strategy, with short term or partial immunosuppression due to age, some other illness or drug. It is sometimes the result of re-exposure. In severe cases, the result is a massive and damaging, but relatively slow immune response. This is not toxic shock syndrome or gram-negative sepsis. It occurs over month to years, with night sweats and weight loss. Fever may or may not be present. Tb is a slow burn. The OP is correct here, that the tissue damage seen in these cases (caseous necrosis, cavitary lesions, sometimes followed by hemorrhage), as well as the night sweats and weight loss, are all caused by the immune response, not directly by any toxin produced by the mycobacterium.
This lung (image from wikimedia commons) was chewed up by giant, angry macrophages, spurred on by CD4+ T-cells:
This lung (image from a public slide set through the University of Utah), on the other hand, was not chewed up by giant angry macrophages, but was lethally infected with M. tuberculosis all the same:
In an immunocompromised host, the clinical syndrome is different. For an HIV patient with a low CD4+ T cell count, alveolar macrophages are not sufficiently activated. There are no large cavitary lesions, but they are also unable to clear or quarantine the replicating bacilli. This results in disseminated tuberculosis, sometimes with spread through the bloodstream to the brain, kidney, and bone, and devastating illness that is universally fatal without treatment. Death may result from organ failure (e.g., of the lung), infectious or non-infectious embolism.
Interestingly, in an AIDS patient with disseminated TB, when you combine TB treatment with HAART (highly active antiretroviral therapy), you can sometimes see a paradoxical worsening of symptoms. This is called immune reconstitution inflammatory syndrome, or IRIS. It may look like worsening TB, with fever and worsening lung function. The solution for TB-IRIS is not immunosuppression, but supportive care and, again, direct treatment with antimycobacterial therapy. Here again, the problem isn't an overactive immune response, it is the pathogen.