I have had the same thing happen to me with comcast internet. There were some times of day where the connection was absolutely pitiful and the IP phone didn't work for a hill of beans!

My ping times had doubled, etc. right around the time I noticed the problem with the IP phone.

This happened for about 2 nights in a row then cleared up. As of late the connection has dropped briefly every morning right at 9:29am on the dot.

From here to the IP phone at my house I normally get ping times that average about 25ms.

From here to the IP phone I have at a salseman's home office in Washington state, I usually get about 83ms. That is over an IPSEC VPN, and he has had no problems. (Knock on wood).