This is a story I have been told as well when I was a kid. Usually this is related to the foreign smell that the humans leave on the chick. However, this seems to be an urban legend, as birds have not a great sense of smell.
Snopes says about this:
However, Mother birds will not reject their babies because they smell
human scent on them, nor will they refuse to set on eggs that have
been handled by a person. Many birds have a limited sense of smell and
cannot detect human scent, or if they can detect it, do not react to
it.
What can happen is that the birds returning to their nest which find the nest disturbed compared to the situation when they left might cause them to temporarily or permanently abondon their nest.
You can find this and more urban legends about birds here and here.