First, the toxicity of any substance is a matter of dose. So the answer of your question is yes too much is toxic but the real question is how much is too much ?
This is still not clear in the litterature but effect are starting to appear over 3% see:
HAYWOOD C., BLOETE M.E. -
Respiratory responses of healthy young women to carbon dioxide inhalation. Journal of Applied Physiology, 1969, 27, pp. 32-35.
OR
ALEXANDER J.K.,WEST J.R.,WOOD J.A., RICHARD D.W. - Analysis of the res- piratory response to carbon dioxide inhalation in varying clinical states of hypercapnia, anoxia and acide-base derangement. Journal of Clinical Investigation, 1995, 34, pp. 511-532.
CO2 is the result of respiration as you know and a too high concentration in the blood result in a change a Ph of your blood and asphyxia. You can find more information on those effect here Permentier and al.
Hope this helps !