One reason you can hold your breath for 30 or more seconds is that you are not denying your body oxygen during that time. Wikipedia says:
After exhaling, adult human lungs still contain 2.5–3 L of air, their functional residual capacity or FRC. On inhalation, only about 350 mL of new, warm, moistened atmospheric air is brought in and is well mixed with the FRC. Consequently, the gas composition of the FRC changes very little during the breathing cycle.
Hence you are refreshing your lungs' air with only 10-15% fresh air with each breath. Furthermore, normal exhaled air has reduced the oxygen to about 16% from the 21% of fresh air. So there is a significant amount of oxygen remaining in the lungs at the end of a normal breath. When holding your breath, much of this remaining oxygen can also be used, albeit at a declining rate due to the reduced amount of oxygen. Note also that when trying to hold one's breath it is typical to fill the lung with more air than usual, thus providing a greater amount of oxygen.
When breathing an inert gas instead of air, the situation is radically different. There is no period where you are receiving any level of replacement oxygen. Normally your blood has some oxygen in it at all times, even when returning through the veins to pick up more oxygen from the lungs. However if the lungs have only inert gases in them, the remaining oxygen in the blood will be diffusing out of the blood and into inert gas in the lungs, making the lack of oxygen more severe. This means that highly-deoxygenated blood will flow out to the body's tissues, where if there is any oxygen present it will diffuse out of the tissues into the blood. This will quickly impact the tissues, and brain tissue has one of the highest requirements for oxygen.
The problem is not that the inert gas replaces the oxygen in the blood as such, it's just that the oxygen isn't available. All gases diffuse between the blood and the gas in the lungs. The inert gas nitrogen (79% of air) normally diffuses this way and is in balance between the blood and the atmosphere. Oxygen and carbon dioxide diffuse too, of course, but they are also helped along by the hemoglobin in red blood cells. The hemoglobin grabs either carbon dioxide or oxygen, preferring the latter. The gas molecules so grabbed had already diffused into the blood. The lack of oxygen in the inert gases in the lungs means carbon dioxide will build up, which is a negative thing but minor compared to the lack of new oxygen for the tissues.