In my opinion, Prof. Allen Gathman's "great 10-minutes video on Youtube" is a pretty waste of time if you already know how hydrolysis happens. In fact, he has not considered the 3'->5' route in an unbiased manner; he doesn't seem to look at the possibility of a triphosphate appearing at the growing 5' tip of the strand in the 3'->5' case.
Actually, the only difference between the two routes (5'->3' and 3'->5') is that the reacting triphosphate appears in different places. In the usual case, the triphosphate which is hydrolysed belongs to the added nucleotide, while in the latter case, the triphosphate which is hydrolysed belongs to the nucleotide on the growing strand. Both are feasible.
In fact, it is known that RNA polymerase has dual activity, but you see, RNA polymerase doesn't have proofreading activity!. Proofreading requires removal of the mismatched base, but in the 3'->5 direction the base's attachment had consumed the triphosphate at the 5' tip of the strand, so it is no longer available to add the replacement base. 3'->5' activity readily destroys proofreading capability of a polymerase So, basically, it is the need for proofreading that restricts the synthesis of DNA strands to 5'->3'. Why it is so, would need a lot more explanation (if in words) but I think a picture has far better explanatory power than a thousand words. I've added a picture from Essential Cell Biology that shows the answer to the 'WHY' question:
The other important consideration is repair. If one or more nucleotide is missing in one strand, repair of the missing nucleotide would be impossible for 3' to 5' synthesis, because no 5'-triphosphate is present. On the other hand, 5' to 3' synthesis does not require a 3'-triphosphate present at the repair site. This is important. That is 3' to 5' synthesis does not allow nucleotide repair.