Sounds like you have network issues.
Run wireshark at one end and you'll likely see delays in the response from the other site.

Don't assume just because you can ping means the network is good. What are the ping times?
Maybe also run ping plotter as it will show you packet loss and latency.