atcomsystems.ca/forum Forums TDM Business Telephone Systems NEC Telephones & Systems Choppy calls SV9100

Hello, set up is super small with the capabilities of this system. But anyway I'm going out on a limb and see if anybody wants to help. I know Toshiba systems and that's it. So this setup is, cable modem -> pfsense -> managed poe switches -> NEC -> NEC IP phones -> SIP trunks also. Plenty of bandwidth, no latency, about 9 extensions maybe two concurrent calls at the most. They had a hack mo of an installer I don't think was certified. It's all running ok but as described by the ppl that use it they on either inbound or outbound calls only the users of the system hear a choppiness or jitter of sorts, the outside caller hears nothing wrong at all. Internal call no issues. I've spent hours looking through packet captures, checking lines, checking internet packets and testing the connection. To pretty much no avail, I once in a while do see packet loss on Wireshark in VOIP calls but pretty much once in a month. They were telling me today once it starts happening the put the caller on hold and then get back to the call and that seems to fix it for that call. No other data runs over this setup besides VOIP data network on a totally different network physically. Other companies in the same area use the same ISP with bigger setups no problem just not a NEC phone system. If any of you would like to help that would be appreciated. I can give any setup info needed I have admin access. TY

Who is the SIPT carrier?
Most require MB 84-10 TOS settings for RTP/RTCP and SIP Trunk. Depends on the carrier.
Usually RTP set as DiffServ 40 and SIP Trunk DiffServ set to 46. A system reset is required after making these changes.

Are you using true SIP trunks or do you have an Adtran or some other device emulating CO lines or a PRI

What Ethernet switches are you using? Are you using remote phones? Are you using a VPN? Are the phones on a completely different network or some sort of VLAN? Is the NEC setup properly for QOS or DIFFSERVE? Has the primary ehthernet port (not the voip port). Been zeroed out?

helpifican ---- voip.ms carrier 84-10 those are set to the typical settings RTP/RCTP | Diffserve | 40 SIP Trunk | Diffserve | 46

mreilly -------- Yes true SIP only cards in this phone system according to the blade config are GCD-CP10+GPZ-IPLE, GCD-DLCA, GCD-COT the analog card isn't used they put that in there for just in case the SIP setup wasn't enough.

Coral Tech ------- Datto Networking E24v3,
No remote phones,
no vpn,
phones, and system are on a completely different network, no VLAN
Is the NEC setup properly for QOS or DIFFSERVE? hmmm, you meaning the 84-10 TOS settings also? or just the PFSense?
Has the primary ehthernet port (not the voip port). Been zeroed out? I'm not sure what you mean

BTW they have Dt700 phones
Thank you, everyone, for the interest.

10-12-01 0.0.0.0
10-12-02 255.0.0.0

If your PFSense or Datto isn't a layer 2 switch DIFFSERVE won't work and you will have to use QOS is the switch allows for it. Also, I am assuming you are G7.11 as codec. Make sure this is fixed. You will also need to check if you or they are using large packets. Seriously though if all else fails turn all this crap off because it may be causing more issues than helping it something isn't compatible. 2 calls 9 phones? Shouldn't need anything and in fact probably ZERO reason to have the networks separated.

agreed on the last sentence, this is the way it was when I got to it.

Only thing on those PFSense does have QOS on. Yes on assumption but weirdly enough incoming SIP calls are set to 729 outgoing is all that's set to 711, but its on both direction of calls and trust me bandwidth is not an issue. Jumbo packets on the switch are currently turned off. I really think compatibility is the issue or some very simple setting or thing in NEC or with NEC is the issue. Also a week before we were using netgear unmanaged switches same issues.

Unless you have a bad VOIP card on the NEC I seriously doubt the NEC is the issue unless something is really programmed wrong. All the codecs need to be in line. SIP ALG needs to be off. I am assuming you are not using a session border control unit. SIP ALG needs to be off on the router/firewall. Codecs must match that's one issue to be sure. Carriers like to compress but sound quality sucks on anything less than 7.11. With so little traffic something basic is going on. I have hundreds of NEC units in the field and I have never seen it an issue with NEC like ever. Seriously consider replacing the PFSense with a different router. ANY router like an ASUS WRT or whatever and see if it goes away. Ya I would simplify the networking and work from there because with that little network, unless you have some real bad latency issues...shouldn't be an issue. Are you running the computers through the phones?

No data on the phone side what so ever purely separated networks. Thats what I mean by NEC wrong, programming wrong in some way.

I think I'm going to start again with PFSense again and switch up the NAT'ing and make sure all is set right. Or also just try a simple netgear router and turn off SIP ALG. SIP ALG isn't a feature or option in PFSense.

So two COMPLETELY separated networks? No common router?

Right 2 modems, 2 firewall/routers, only commonality same ISP.

Make sure if you are using NAT you MUST define your local network scope in 10-58. ie IF YOU ARE USING 192.168.X.X use 192.168.0.0 subnet 255.255.0.0. If you do NOT define this you will have routing issues on the system.

These are not filled in currently.

I think I know where the codec setup is for the SIP trunk 84-19. 15-05 15 - Codec Type [type1] where is type one defined?

https://www.intelepeer.com/wp-content/uploads/2018/04/NEC-SV9100.pdf

Look at some this it may show you. Every SIP carrier is a little different.

Thank you, on monday I will check the 40 or so things that maybe wrong after hours there.

You might want to see if NEC has a white paper on the company you are using for SIP. There are a TON of them because everyone is a little different.

Yeah, there's nothing I even asked the SIP company directly if they had something, they have like 10 config guides for other PBX's not NEC though.

Just an idea. Not like NEC isn't a major player just your SIP provider hasn't bothered to get NEC certified. That being said choppy calls means they are establishing so look deeper. AND not to be a debbie downer you are not using ANY sort of MPLS or QOS service so there are no guarantees.

I added 10-58 Set local network area of DT800/DT700, the correct ip range 10.0.0.1 sub 255.0.0.0 yes 16 million range for 10 devices(don't ask) .

Added 84-13-28: Sip Trunk Codec Setup - Audio Capability Priority to G.711_PT

So all codecs are running the same.

Those changes made no difference

Yeah it's something to do with the router/routing I think PFSense had QOS running in its software its called Traffic Shaping. Two pieces to it actually bandwidth throttle and prioritizing types of traffic.

What I did for testing was to get a baseline before any changes. I would run one or two outbound calls to a number that automatically started talking recordings 858-651-5050 if you ever need it for any reason great way to test or go insane whichever better then calling up your wife and making her help you.
Within 10 minutes I would get a choppiness that was really bad, the timing was random would be at minute one or minute 4 or whenever it would be bad for about 30 seconds then clear right back up.

with that at least pinpointed that routing/reshaping of packets or the ISP connection(highly unlikely) is the problem because of VOIP/RDP when packets get out of order because of timing instead of sorting them or resending it will just toss those packets out causing the choppiness. Getting this big of an issue because the router is was either set incorrectly or too many QOS rules being applied. I really need to remove the pfsense out of the picture for a good test.

I disabled all the QOS'ing on pf as best I could but I think it's still imposing something even though I can check monitoring graphs of the QOS and in real time on it and see nothing happening. I believe it either remedied it or made it like 80% better in my testing.

Why 255.0.0.0 for the subnet mask?

Yep, great question have no idea what the person who setup the network was thinking. 16.7 million range. It's been easier to just leave it alone though.

By all means I'm not a IT expert, but my understanding is the subnet mask of 255.255.255.0 speeds up the networking process, 255.0.0.0 slows it down.
Why would it be difficult to change it?

No subnet is just what predetermines your ip range and "Mask" the rest. So no not really. I would have to change firewall switches and PBX I really don't need to mess with the unknown just yet. Just fixing a choppy audio problem. It would be nice to someday so if I do an ip scan or something for whatever reason the tool doesnt have to run through 16.7 million possible used addresses. But switching gear certainly now and days to not scan entire address ranges for packet delivery to slow down anything at worst the sending devices sends to broadcast for a helping hand for delivery. It's just stupid though for managing. 254 range is far beyond enough. only ppl that need 10.0.0.0/8 range are company's handling millions of devices ISP's and datacenters.

Yeah, there is NO reason 192.168.x.x, 172.x.x.x, 10.x.x.x shouldn't be enabled by default. This simply makes the system route internally. If you have ANY IP in the external IP address on the main processor for NAT you must do this or you will have issues. Ya go get cheap router and put it in and see if it goes away.

Hello, tried a cheap router a Netgear something. I set the network for it correctly, turned off SIP ALG and opened the NAT. Also, put the MAC address from other routers WAN port to the Netgear routers WAN port for the static IP it's given. After that connected to the SIP provider just fine and made received calls just fine, but about 15 minutes or so in, the audio would drop this was an outbound call to an 800# this was repeatable I was not talking to a person so I do not know if it was both ways, but the call control was fine the PBX did not drop the call completely just the audio went.

I'm pretty sure I have to port forward/trigger, Netgear's wordage, on the RDP's ports. Just to make sure that's not the cause. Question mark maybe.

You need to wirshark this. Make sure you check and see that UDP flood protect is off. Some routers/switches do not like seeing that traffic. Seems weird if happens after a bit. Wireshark is needed to see what is causing the drop. The router should NOT being doing the NAT control your PBX is doing that. ALL you need are the ports forwarded and get the heck out of the way. Also, check your keep alive timers.

K thank you.

Hello, using the pfsense, there is no UDP flood protection on as far as I know if wanted those types of blocking would have to be turned on. I look over multiple wireshark captures from the lan side and wan side I see nothing crazy. RTP data looks super clean.
I'm going to clean up the pfsense yes it has NAT on to public IP to the phone server. I don't know why. I'm going to get rid of that and have PFsense just unblock necessary traffic in its firewall rules only and everything else settings wise its got for messing up VOIP/SIP traffic I will turn off or set to the max for making the connections better. I think that will be the big helper for this issue.
The issue is once again at some random time in an outgoing or incoming call locally you will get a pretty annoying "drop"/mismatch of packets = to the outside caller is hard to understand for about 10 seconds then it comes back. You would think the call is about to drop, but it doesn't. I guess they were also saying they can put the outside caller on hold for a second and then bring them back and it clears it up. It screams out of order packets, but wire shark does think it is.
Heres just one of the rtp analysis for one 13 minute call. Another sad thing is I really can't get much traffic to spy on it's so small of call volume.
SSRC
0x38e8ac40
Max Delta
25.70 ms @ 86
Max Jitter
0.74 ms
Mean Jitter
0.11 ms
Max Skew
-4.96 ms
RTP Packets
509
Expected
509
Lost
0 (0.00 %)
Seq Errs
0
Start at
2.086294 s @ 4
Duration
10.16 s
Clock Drift
-34 ms
Freq Drift
7973 Hz (-0.34 %)

Reverse

SSRC
0x856f8885
Max Delta
20.80 ms @ 44998
Max Jitter
0.18 ms
Mean Jitter
0.06 ms
Max Skew
-8.40 ms
RTP Packets
39963
Expected
39963
Lost
0 (0.00 %)
Seq Errs
0
Start at
2.103154 s @ 5
Duration
799.25 s
Clock Drift
-8 ms
Freq Drift
8000 Hz (-0.00 %)

You know, are you getting "jumbo" packets by any chance?

Not set on by default on the switch so i would think they stop there. Also if I take out NAT on the router, I would just turn off 10-29-21 and that's it correct?

You know you should consider getting a free sip line from another carrier just to test baselines. This could be entirely them.

Hmm, that's true.

From what I understand they have tried that in the past not too sure on that. I still will try after I get the routing correct in the firewall.

Ya, seriously without an MPLS or QOS connect you could be testing this all day and it's beyond your control. The only thing you can do that may help is going to 7.29 compression even then you have no QOS from point to point. I run SIP all the time from carriers and like hardly zero issues as long as I have a QOS or some type.

Hello, I totally forgot that I did end up figuring this out fix a stupid easy fix apparently at some point in time the IP card on the phone system had been set to 5mp half duplex got it to switch to 1gb full and all was well. I found this by looking at the managed switch showing status of connection and seen it before I originally posted but must had thought it was the way it was supposed to be. Thank you for everyone's help