atcomsystems.ca/forum Forums TDM Business Telephone Systems Toshiba Telephones & Systems Remote IP phone users sound "under water"

Hello, not sure exactly what to post, but I am at my wits end and may need more advice than fixes, but a fix would be great.

About 8 months ago we went live on a new Stratagy CTX 1000. This was an upgrade from an older system and some cards were brought over from the old system so it is not 100% new. We have the IP card and have 5 remote phone users all with brand new Toshiba IP phones.

Without airing all the dirty laundry the project was terribly managed by the dealer and while they knocked out the problems over time this one has never been solved. I do not beleive the IP part of this is set up correctly as they were zero help in configuring the remote phones.

First it was that I didn't have enough bandwidth, but I have plenty according to the one knowledgeable tech who actually made an improvement (full t1). Second it was that the card had to be bad, supposedly the replacement card was ordered but I have not been able to get anyone to come out and actually put in a replacement. Part of this test included hooking up an IP phone at their office and they tested it and duplicated the problem. Now they are telling me there is no longer a problem and there is nothing that can be done since they cant find anything wrong.

Except for the fact that I get daily complaints from my customers about people they speak to being described as being under water, gargling, and just plain choppy and breaking up. Keeping in mind this is ALL remote people, not isolated, I am at a loss.

It might be as simple as getting another Toshiba dealer to look at it, but we have had terrible luck with the Toshiba dealers in our area and I have no confidence. I also dont want to spend a ton of money to fix a problem my current dealer should have found and fixed under warranty.

Any suggestions are appreciated!

My first question would be....Is the T-1 a full 24 channels dedicated to the phones, or is it shared with their data network? If its shared, is the router set for QOS?

Is the T-1 a data T? What are you using for phone lines?

Each phone takes about 80-110 kbps, so for the purposes of easy math, all 5 phones would use about 500kbps, or about a third of your T1.

I don't know what kind of data you are passing, if you are steaming the presidential debates or stuff on youtube.

The problem could be that your phone calls have to fight with everyone else for bandwith. It could help a lot if you had a router that supported QOS. Once you hit the Internet there is no QOS, but you need to prioritize the voice traffic out to the WAN.

Are all 5 users at different locations? Sometimes it can be the internet provider. I usually start troubleshooting by using a continuous PING from end point to end poing. Delays should be under 200ms, or else you've have problems with quality.

You can ask your vendow if they have the latest firmware on all the equipment.

I've PM'd you, fixter.

Thanks so much for the reply.

It is a shared T-1 that scales voice with data over 12 channels and 12 are reserved for data only. All main office phones are NOT IP.

We are only 12 people in the main office and the major source of internet traffic is RDP (terminal services) from remote users and from using gotoassist remote control software on and off by 2-3 users at the main office throughout the day.
Based on testing of bandwidth usage I dont believe that we are even coming close to using it all.

We are not using QOS, however they bypassed the router when the set up the IP card. All VOIP traffic is off my internal network. So I had considered QOS when I was researching myself, but since we bypassed the router there was no benefit.

The users all have different ISPs. They also all have different routers at home. We asked before we went live with this if it mattered and we were told "any router will do." The only reason that I didn't question this was because all 5 of these people had been using vonage for 2 years on the same equipment without incident. This is the biggest reason I think the problem is here.

I really appreciate you guys taking the time. I have been a long time lurker here and always find the answers I am looking for.

Since you seem knowledgeable about bandwidth/networking, take a phone to your house, turn off all unnecessary services and make test calls to your office during and after business hours.

Have done that and replicated the problem. What I dont know is if ther are settings in the routers at the remote locations that need tweaking? Also hooking up the IP phone internal to our network is crystal clear on all calls.

Well, not any router. Try using a new Linksys WRT54G router and the phones won't even connect.

All 5 users had vonage? Does vonage use a router specifically for their service? Just wondering if that was interfering.

You are right that QOS is usless before the router. I have setup systems like that many times typically without a problem.

what kind of service do they have at home. i dont install ip phones with dsl. DSL is to up and down. Make sure that it is cable and above!

Have them replace the cards ASAP. If they replicated the problem and admitted it, they need to replace part or all of your hardware. The least they can do is to send a tech to troubleshoot. Setting up IP phones on a public connection is simple enough. I don't believe they would have been able to mess up that part. Also, did you have them change the codec? or packet transmission interval via voice packet table?

3 of the 5 have cable, so I am thinking it isnt a DSL or cable thing.

I finally got them to agree to come out- so we will see.

Any time we've had trouble getting connected, we have the customer purchase the TRENDnet Wireless G router . The problem you're describing sounds like a programming issue though. Hopefully the installing dealer can duplicate and fix the issue.

if it works crystal clear with ip phones internal than id say cards and programming are all ok...

I am not a big fan of those shared T1 circuits... that is probably where your problem resides,

I bet if you got a cable modem with static ip dedicated to the cix phones only you would be ok

you shouldnt be so quick to crucify your dealer when the problem has to do with the internet and the type of circuit coming in, and not the system

I would say 8 months isnt quick to crucify my dealer.

As far as the shared T1, other than "not being a fan" can you tell me why you think that is where the problem lies?

All bandwidth tests indicate we are never getting close to utilizing all of our capacity.

The main issue I've seen with those shared T1s is with the voice side. Many care the voice with VoIP, and often use compression. Usually sound quallity is ok, but faxing and modems connections are very unreliable at best.

We are having the sames issues as fixter. We have a T1 for our PRI and a separate T1 for data and external IP phones through a switch. We tried a priority router and gave 100% priority to the IP phones. That made no difference. Tomorrow, we are trying a separate cable modem for the ip phones. I hope that works. Otherwise, I guess we blame the internet. But, that would make our ip phones obsolete. Can that be? What a waste of money.

just turn DiffServ off. Especially if you are using Cable providers at the house. I have run into issues where I had crazy type of problems with that. No voice, phone connects rings, but no voice. Bad voice quality... an so on.

I am working on it with my toshiba dealer tomorrow. He has tried to fix the issues for 2 years. They blame it on the internet. Can I find DiffServ in Emanager?

At home I use Verizon FIOS which is like DSL, I guess, 10 down 2 up. At work, the dedicated cable modem will be 5 down 800K up.

I have seen cases where giving the IP phones there own internet fixes issues. Best tool to do some basic troubleshooting s a continuous ping from end point to the IP card. If you get replies over 100ms, then you have call quality issues.

How are you prioritizing traffic? You have the IP card behind the router with the rest of the network?

Difserv is one way to proritize, but as TadiranTechTexas said, it may not play nice with cable internet. I have often seen no or one way audio when turned on.

With T1 (at office) into switch out to static ip at phone and static ip at firewall (network), ping from home computer on verizon fios to phone ip address is 81ms average at 10p. Could it be different with internet traffic during work hours?

Now we are not prioritizing, but when we did try that, we used a router that gave up 100% priority to phone ip address. That made no difference.

Phone ip card is after switch that T1 goes into. Out of switch are 2 ethernet cables. One to firewall-network and one to phones. They each have their own static ip address.

What issues could I have if Diffserv is disabled?

Weird, ping to our default gateway is 21ms average.

Since you are basically going over the internet, Diffserv won't make a difference of call quality over the internet but can cause issues if it is turned on and the Cable companies are not configured to look for it. Diffserv would play a big role if you had an AVPN or MPLS network topology with multiple sites. It would give priority to the voice packets.

cable modem attached to phone system or at remote ip phone sites? Does cable include verizon fios, dsl type service?

I don't know how useful this will be but here at my facility we're using G711 codecs primary and G729A secondary for for phones right here on the campuses, and for remote users I have G729A as the primary and no secondary selected. I've had good luck with this setup. You'll want to use voice packet table 2 with G729A. G711 would probably be a bit too heavy for a remote phone over a DSL or Cable connection in most cases.

The setup at the office is typically how I have set it up. The IP card needs a public IP address to work, but since it is outside the router it makes it difficult to have a router proritize traffic.

Ping of 81ms from end to end is pretty good, if it is like that all the time. Last time I was trying to troubleshoot call quality, I had very good ping results most of the time, then there would be spikes of like 500+ms. I was trobubleshooting on the IP phone side, but the delays were on the phone system side.I believe it was some server that would burst a large amount of data periodically, and that is when the calls sounded bad.

You can definately try using G729A to see if it works better. If delay is a problem and not bandwith, then it could sound worse.

In the office now and have IP phone with me. IP phone is connected to office network, phone system is on own cable modem separate from network.

ping from work station connected to ip phone was 58 ms average, but at other ping attempts, there were timeouts and loss 25%. I spoke with someone on the ip phone for a while. Every so often, I could not hear them and then call got disconnected eventually. Good quality most of the call. Call was to another digital extension in my office.

Diffserv is still enabled. My toshiba guy did not want to disable it.

The cable modem dedicated to the phone system is 5 down 800K up. Is that enough, or do i need to bump it to 10 down, 2 up?

You can have him at least try it and if it doesn't get better or fix it you can always turn it back on.

Thaks for the reply. I think I can do it in eManager. Is it in IP-Telephony-->system IP data assignment-->03 Diffserv disable?

Are there any other settings I can and should tinker with? Audio codec for remote IP phones are G711 with secondary audio codec for full IP station at G729A. Any others?

Using cable modem at one remote IP phone and verizon fios at another and cable modem at the system with 5 down 800K up.

Rick,

You said that on your remote IP phones you're using G711 as the primary codec. I'd go with G729A for the primary and no secondary. Select voice packet table 2. It's just a lighter codec for use over an internet connection. Unless of course, you've got some kind of crazy fast pipe on both ends. I also agree with TardianTechTexas..we don't even have Diffserv enabled at all where I'm at. Try turning it off.

Do yourself a favour and check the firmware version of the IPU card.
Also, which card is it? BIPU, LIPU, or MIPU?
Also upgrade the firmware on the sets.
Also download pingplotter. Run it for a while when you are having problems. It will show you which router hop is being the bad boy (saved my butt a bunch of times).
I had an LIPU that was causing me grief, I thought it was internet related, ran pingplotter, connection pretty good, upgraded the firmware and presto, problem gone!

Dave

LIPU_02_03D

Thanks for everyone's help. I have disabled diffserv, changed the phones to G729A and none for secondary on the phones and packet table 2 for the phones. I'll see tomorrow how people respond on the other end of the calls.

Canuckvoip, may be a little too much for me to upgrade firmware. We had the system installed 9/07 and in 8/08, the vendor installed the latest lipu card and software, I believe. I don't want to mess with that. We also had Toshiba involved. They blamed it on the internet just like our vendor.

I am deffinately going to try pingplotter, at least the freeware version. I love having a large arsenal of tools. Most are based on some type of ping or traceroute.

It is very common for Toshiba and vendors to say it is the connection. In most cases that is where the problem is at. Sometimes the internet connect just won't support a VoIP calls because of the delays or jitter.

Packet loss of 25% is pretty bad. There is definately a problem there. Sometimes I wll open another window and ping google.com as a baseline to see if the connection problem is closest to the side I am on. You can also ping the remote gateway. Since the system is on it's own internet it sould not be a problem with data traffic fighting for internet, unless you are downloading windows updates at the phone side while trying to talk.

Sometimes updating the firmware will help. The latest LIPU firmaware is 02_05D. It may be a bit much for the end user to update the firmware. That is why it is recommended for a Toshiba dealer to do it. If things go bad then they can call tech supoprt, while you can not.

I am a little puzzled about your Toshiba guy not wanting to disable Difserv. In your setup I see no benefit. If the devices on the network do not support QOS, the header is usually ignored, but there are rare occations where it can cause problems.

This type of troubleshooting is never easy.

So, after 2 years of awful voice quality on our phones, so far so good today and ping to phone has been 20-30ms.

Yesterday's changes include: dedicated cable modem (5/800K) for phones system, disable diffserv and ip phones get - packet table 2, g729a codec and none for secondary codec.

Fingers crossed for future success.

Thanks to all, so far.

What do g711 and g729a represent and what are the differences?

Compression ratio's. g711 is a higher compression,uses more bandwidth and usually gives better voice quality. g729 is a lower compression, uses less bandwidth and in some cases will provide poorer voice quality.

These represent compression levels. G711 requires a lot more bandwidth than G729. If you have the bandwidth available, the using G711 is preferred since there is less compression, thus less room for errors. G729 compresses the data packets more, requiring much less bandwidth per call. If the connection isn't ideal, there is the risk of unsatisfactory voice transmission when using G729.

Crossing paths again with replies. Sorry about that.

Thanks. Trying to figure out why my dealer put codec at g711 and g729a as secondary and this forum recommends g729a only and no secondary.

I have cable 5/800K at system and fios 10/2 and cable 5/1 at remote phones.

Well I suggested that because that is a setup I've had very good success with. I have two remote IP phones. One VERY far away in Washington state, and another at my home 30 minutes away. We're talking a lot more hops to Washington, but the voice quality is the same. I've used both with G729A. If you mouse over the codec in eManager, it actually suggests using that codec for remote IP phones using a DSL or Cable connection.

The dealer probably used the default settings. I believe when you set up an IP phone for the first time in eManager it almost always gives you G711 for a primary and G729 for a secondary codec. That could be your answer regarding that one.

Thank you again. This is the 1st time I have been able use the IP phone in 2 years of having the system. I used it all day without issue. I hope that is an indication of future use.

My boss is thrilled. I will not be sharing this info with my dealer and they will no longer be my dealer. Toshiba got involved a year ago and they were worthless too.

FYI...G.711 is the default compression algorithm on a Toshiba system.

IP phones worked well for the day yesterday, but at 6p, I heard alot of garble then phone went dead and came back couple minutes later. Internet at my house on same router was still up though and phone system modem was still up.

Any other suggestions on disable/enable?

Download/install/run pingplotter.
Leave it running/pinging the IPU card.
Voip is a realtime protocol (RTP). Most likely either your or the IPU end had severe delay or packet loss. Computers don't care cuz if a packet is lost, the other end asks for a resend.
Not so with voip, no packets are ever resent.
Leave pingplotter running ALL THE TIME at about 2.5 sec intervals. When you experience the problem again (and you likely will) you will see which router hop is being bad.
Were you or someone else at home downloading large files at the time? Maybe a bunch of Windows updates at the IPU end?

Dave

6pm is a bad time for open internet calls no matter where you are. Everyone is getting home and jumping on the net. Modem would still show up because it is more tolerant of bandwidth issues.

Tadiran,
Absolutely right. Even dsl users once the packets leave the providers' domain are subject to anything and everything going on in the internet.
Voip has issues with delay and jitter, but packet loss is a killer!

Dave

It's funny cuz I always figured the Internet was busier during working hours, then at night.

Since phone have their own internet, and you or someone in your home isn't streaming a new music video, then it could be the internet provider.

We had to take back an IP phone because we just could not get the one phone to work at all over a DSL connection. The bandwith it just one factor. Delay and jitter are at least as important considerations.

I downloaded the free version of ping plotter. That looks like a very usefull tool to track connection problems. I would keep that running al the time, then look at it if you have a problem with a call. It doesn't look like the free version will keep logs or statistics. I didn't check to see if the other versions had a time trial.

newtecky,
yep the pro version has a "vcr" type function to see the history from many hours ago.
Pro version is 30 days I think with nothing hobbled.
Will give you jitter and a voip mos score as well. Best $150.00 i ever spent.

Rick,
It sounds like you licked a hardware/software problem. Let's look at the internet again. I know you are rolling your eyes after 2 years but hear me... seriously...

Dave

Have been running ping plotter and am seeing some packet loss and some latency at certain hops. What do I do with the information?

now getting substantial packet loss consistently at the phone system and remote phone has disconnected itself.

If it is at certain "hops" then there really isn't much you can do since you are at the mercy of the internet providers. Possibly put a VPN connection between the remote site and the main site, at least you can try to control the packet loss that way by have a VPN tunnel nailing up the connection always instead of just the keep alive packets going every few times...

How do I set up a VPN from remote phone to system?

I did try to set up a vpn from my home computer to the phone system. Didn't work?

I also work through a vpn to my office, but that's to my office data network, which does not contain the phone system.

Pingplotter will, hopefully tell you were the problem is occuring. If, for example, you are getting timeouts at the phone system IP card, but the hop just before that one is looking good, then you have to look closer to that side. If the gateway to the phone system looks good all the time, thing maybe something internal, like a data switch, or the IP card itself.

If the problem is somewhere towards the middle, then it is definately a problem with the internet connection. Not sure how to solve that one, except to try calling the provider and seeing if they can help.

You are seeing ping results from the connections on the way to the phone system. Rememeber you are looking for where the connection problems start.

In order to create a VPN for the phone system, you would need a router on each end to create a tunnel, which means you would have to put the IPU on the phone system behind a router, then everyone that connect to it would need to use a VPN.

And, is 5 down 800K up enough cable modem speed at the phone system?

Hi Rick,
G711 phone calls take about 100k.
G729 (compressed) take about 40k.
Because your upload speed is 800k, that is your effective limit (there is no use being able to hear if you cannot speak, right).
Also, nobody ever gets what they bought speed wise.
So, if the connection was used FOR VOICE ONLY, you'd be OK up to 7 simultaneous phone calls at G711.
Obviously, if the connection is shared, the computers will hog as much bandwidth as possible without some form of traffic shaping or Qos.
I don't know how many phones you have or what your computer needs are so can't really say if 5/800 is enough or not.
One thing is for sure though, if you are experiencing lots of delay and packet loss, increasing the bandwidth will do NOTHING!
I have had success in the past taking the screenshot from pingplotter and email/showing it to the ISP if one of the hops looks bad.

Dave

I have had the same thing happen to me with comcast internet. There were some times of day where the connection was absolutely pitiful and the IP phone didn't work for a hill of beans!

My ping times had doubled, etc. right around the time I noticed the problem with the IP phone.

This happened for about 2 nights in a row then cleared up. As of late the connection has dropped briefly every morning right at 9:29am on the dot.

From here to the IP phone at my house I normally get ping times that average about 25ms.

From here to the IP phone I have at a salseman's home office in Washington state, I usually get about 83ms. That is over an IPSEC VPN, and he has had no problems. (Knock on wood).

I think I saw earlier in this thread people arguing the virtues of g729 over g711.
We ONLY do g711 unless otherwise needed.
Why?
In either case, the size of the packet is the same. But, in the case of g729 there is MUCH more info crammed into each packet. Therefore, if you are experiencing packet loss (anything over 3-5% is deadly), you will experience WORSE call quality and/or more frequent disconnects.
Whomever it was that suggested getting rid of diffserve was probably the saviour as that is a waste of time and packets on the internet.

I have seen some modems having a problem and rebooting the modem (at the enterprise end) can fix it temporarily.
If you continue to have packet loss problems get on your ISP's case and don't take no for an answer!
Again, if your packet loss is over 3-5% you will have problems but will have fewer of them on g711.

Dave

Yep. I was puzzles when rglassband said his Toshiba dealer wanted to keep Difserv on. Useless,a nd even a problamatic when going over the Internet.

A lot of people recommended changing to G729. I have tried to do a lot of things as a test, such as make extreme changes to jitter buffer, ect, and even change to g729. But that will only help if the issue is bandwith related. Both you and I have mentioned delay and jitter a couple times.

Other techs have said that they have had more trouble with VoIP over cable modem then DSL, yet cable always seems to provide much more bandwith to the customers then DSL.


Toshiba TCTE
CTP

so, diffserv off and use g711 as primary and g729 as secondary codec? Specs - Phone system on 5/800k dedicated cable modem and remote phones on 10/2 verizon fios and second one on comcast cable modem 5/1.

Which primary and secondary voice packet tables?