Great question! This apparent contradiction has puzzled many neuroscience students before you.
This is often called "shunting inhibition," in particular when excitatory and inhibitory conductances are out on dendrites.
The part that is wrong is this (emphasis mine):
the internal voltage will increase, hence coming closer to the action potential. This would increase the probability of reaching an action potential, not decrease it, wouldn't it?
The idea that "hyperpolarization is inhibitory, depolarization is excitatory" is only partly true. It is very important to also consider what the spike threshold is, and to think in terms of reversal potential for a given ion channel (or more generically we can just call it a 'conductance'), and then you can revise the statement to say this:
Conductances with reversal potential greater than spike threshold are excitatory, conductances with reversal potential less than spike threshold are inhibitory.
Most often, chloride-based channels fit into the second statement: their reversal potential is less than the spike threshold.
How Shunting Inhibition Works:
Any time you open a channel, you will shift the membrane potential towards the reversal potential for that channel. The amount of current that flows through a channel depends on the 'driving force': the difference in voltage from the reversal potential.
Let's consider a fairly typical textbook cell, with a spike threshold at -50mV, a resting potential of -65mV, and a chloride reversal at -60mV.
If the cell is at rest, and you open chloride channels (such as with GABA via GABA-A receptors), the resulting flow of current will tend to push the membrane potential towards -60mV, so the cell 'depolarizes'. However, no matter how big of a chloride conductance you open, you will never pass -60mV, so you will never reach spike threshold.
If, in that same cell, you instead opened AMPA channels, with a reversal of around 0mV, you also get depolarization, but in this case as you open more AMPA channels you can potentially depolarize the cell all the way to 0mV. Of course, unless you have blocked sodium channels you will get an action potential before you reach that point, but that's the key: you will cross threshold, therefore we call it excitatory.
Now let's consider a third case where we open both AMPA channels and GABA-A channels. As long as the membrane potential is <-60mV, both channels contribute to depolarization. However, as soon as the membrane potential is >-60mV, chloride ions will start flowing into the cell. We call this "shunting" inhibition because if you look at the sum current flow in the cell, it will look small, because you have chloride ions coming in at the same time as sodium ions, resulting in little change of the membrane potential despite lots of ions moving.
The result is that GABA-A channels, even if they can depolarize a cell that is at rest, will act to prevent the cell from depolarizing far enough to reach threshold. It's important to consider the dynamics of the membrane potential, rather than thinking of the membrane potential as something that is simply added or subtracted to instantaneously.
Ion concentrations matter! If chloride concentrations are not 'typical' or if the spike threshold is more negative than in a 'typical' textbook cell, then chloride conductances can indeed be excitatory! In fact, excitatory GABAergic transmission is important are certain stages of development. If cells have too much chloride in them, this can also cause GABAergic transmission to become excitatory (or at least limit the efficacy of the inhibition), and this can lead to epilepsy (see Cohen et al. 2002).
There can also be a time window where GABAergic transmission is excitatory even in a typical cell: this is the period where the GABA-A channels have closed, but the membrane remains slightly depolarized and has not returned to rest. Excitation that arrives during this time will sum with the residual depolarization, and there are no open GABA-A channels to 'shunt' the current.
Alger, B. E., & Nicoll, R. A. (1979). GABA-mediated biphasic inhibitory responses in hippocampus. Nature, 281(5729), 315.
Cohen, I., Navarro, V., Clemenceau, S., Baulac, M., & Miles, R. (2002). On the origin of interictal activity in human temporal lobe epilepsy in vitro. Science, 298(5597), 1418-1421.
Purves, D., Augustine, G. J., Fitzpatrick, D., Hall, W. C., LaMantia, A. S., McNamara, J. O., & White, L. E. (2014). Neuroscience, 2008. De Boeck, Sinauer, Sunderland, Mass.