I'll borrow a diagram from another answer to a similar question, but not quite a duplicate (I hope @AliceD won't mind my laziness):

Fig. 1. Refractoriness. source: University of British Columbia
One important concept to understand about the refractory period is that it has two components. One is the hyperpolarizing part, which is due to open voltage-gated potassium channels. Because there is more potassium conductance than at rest, the membrane voltage gets closer to the reversal potential for potassium.
If you were to open a sodium conductance during this period, you would indeed get a larger sodium current than you would get at rest (because you are further from the sodium reversal potential), however it will also be accompanied by potassium leaving the cell (because there is more potassium conductance), such that the net current and net change in voltage may not be much larger than at rest. Voltage-gated sodium channels that open to cause an action potential are gated by voltage, not by sodium. However, this type of refractory period is referred to as a relative refractory period because you can overcome it if you open a large enough sodium conductance (or inject a bunch of current with a patch clamp amplifier).
However, there is also a second part to the refractory period, the "absolute" part, which is shown in the diagram above. The voltage gated sodium channels themselves are not only closed but they are also inactivated. While inactivated, those channels will not open even in response to depolarizing voltage. It takes some time for the channels to return to the 'closed' state where they can be opened by voltage.
Summary
To speak to your question more directly, nothing prevents sodium from diffusing 'backwards', but action potentials travel quickly, sodium channel inactivation prevents them from traveling into axon segments where the sodium channels were just recently open, and the open potassium channels that cause hyperpolarization will 'shunt' current out of the cell while they remain open. Lastly, voltage-gated sodium channels are gated by voltage, not sodium.