If there are bats flying in the cave, and I play the same pitch they use to echolocate from a speaker, what would the bats do?
Depends completely on how good ultrasound you are producing.
Will they interpret your signal as an obstacle ?
Not really until you produce signals exactly calculated for the distance and speed of bat from obstacle. Here something known as dopler effect will come into play which will make bats understand probably thats not their sound. Otherwise every bat would have made a noise to divert another bat.
Is just making sound good for dislocating
Well, till this date we don't know what humans are speaking from their sound waves what you can guess for bat for what they did just speak. If you listen closely to bat sound you will find amplitude variation at non-constant duration.
Here is a detail of what else bats use . Its harmonics that takes sleep of physicists.