I have a customer using an XTSc in their main office and about ten IP phones located all over the country. Lately, they have been experiencing various phantom problems with the IP phones. Most of the issues have been attributed to the Internet connections at the far ends, but this one has me stumped. They claim that X131 (IP phone) is experiencing crosstalk (yep, I didn't stutter) with digital X107. This pretty much confirms that the cause must be within the system, but I can't prove anything.

Also, I noted in a CONFIG dump that all of the IP phones were showing as being in-use on various PRI channels even though they weren't. I had to physically release each station first, then the PRI channels after that in order to get the system to show as idle. This has happened twice so far.

I've replaced the CKIB (usual culprit), the IP card, various phones and even the entire KSU, but the problem remains. They are currently using 4.00 with ACD. Any suggestions?

3.0 or 3.5?

Overloaded XTSc? It can only work with 28 trunks. How are you managing with a PRI and a Lan card and four pots? That is at least 28 trunks if it is a full PRI? I would never trust a c to work with that much load.

Yeah, Derrick. We are using a clustered PRI with twelve channels and twelve on the IP card plus the CKIB. I know it is loaded, but we're going to move this XTSc to a remote office for use as a node and replace it with an XTS at some point. Here's something odd that I discovered today while grasping at straws:

When we upgraded to FP4, we changed out the whole KSU to minimize downtime. After this, I found one of the IP phones no longer communicating whatsoever. I decided to check the MAC address as verified by the end user. The MAC he gave me was nowhere close to what was in the system, so I changed it. After this, we realized that he had given me the wrong MAC, so I put the original one back in.

Voila! The phone started working. I have a suspicion that the upload/download process must have a few bugs in it. I'm going to try this with the other three extensions that are having inexplicable problems tonight to see what happens.

I know that by rights, the MAC can't be "just a little bit off", so the phones should either work or not. Since this seemed to clear one problem, I'm not ruling anything out.

Update: I went to check the system and to re-enter the MAC addresses. Once again, I found seven IP stations showing in-use and had IP trunks 5-11 tied up. I had to reset the card to release them. Even though we are using a 12 port card, we only have 8 of them active. They have also been complaining of "no IP trunk available" displays.

What appears to be happening is that when an IP phone establishes a call, it remains married to that IP trunk permanently until manually released. I'm starting to think that the phantom problems might be occurring when a ninth IP phone tries to make a call. In addition to crosstalk, we are also getting dropped call complaints.

It turns out that we have 8 IP trunks designated, 16 PRI trunks clustered and the four loop-start COs on the CKIB. With 16 trunks available for incoming calls, and with ACD in use, I can see where more calls can hit than we can handle via IP phones. Most of the ACD stations are IP sets.

Does that add any more brain food?

It sounds reasonable that you could have a database problem. Twice I have had Ip phones go offline on an XTS. The display was still there but when you called the phones you get nothing, no fast busy, no ringtones, and the digital phone that called the ip phone would lockup. The only cure was to power down the system and turn it back on. Oh and a config ip card slot dump showed the phones in service as well.

What if the acd sends a call to an ip phone and no ip trunk is available? I would think the system might burp a bit if that happens.

Funny you should ask. I just sent this response to the customer and was thinking more as I typed. Take a look and see if we might be onto something:

OK, I've reinstalled all of the MAC addresses tonight. I must say that I'm concerned that once again, I saw that seven of the IP phones were engaged, yet no conversations were present. It looks like they "married" themselves to IP trunks for some reason. I had to manually release them again.

We do have a 12-port IP card, but only 8 of the ports are active. Since the cabinet only allows for 28 trunk ports, we have 16 as PRI channels, 4 for POTS lines (cannot be removed) and the 8 IP channels. What appears to be happening is that we are approaching the upper limit of the cabinet. I feel that the occasional ninth call attempt is what is causing the erratic operation or dropped calls. Although this shouldn't happen, I can't help but wonder since the failures are so random.

Although is it not mandatory to have a channel per phone, we do need one per active call. Theoretically in a normal environment, we can get away with a 2:1 station to channel ratio. Idle IP phones do not require a channel, but active ones do. In an ACD environment, I suppose that 9+ calls can be presented. I'd think that the ninth call attempt would just not go through, but it looks like this might be A) Bumping an existing call off; B) Causing a collision where crosstalk might occur or C) Prevent other stations from being able to connect.

This "marriage" would also explain why some IP phones occasionally can't connect. If eight of them lock onto the eight ports, then any ninth phone won't be able to connect. Since I really don't think I did anything to cause X118 to magically connect, me thinks that an IP channel just became available to allow it in. I think I'm onto something.

If my suspicion is correct, we can take away four of the PRI channels to allow four more for IP. The only problem with doing that is the fact that we'd be reducing the ability to have sufficient incoming/outgoing call handling capability. Of course, this may mean that we need to start thinking about bumping up to the larger cabinet. I know we discussed the possibility of doing this and using the smaller cabinet for a node in Ketchum, but this may need to happen sooner than later.

I'm not implying that we should jump into this. Perhaps the next step we should take is to change the port configuration as I described temporarily to see if the IP phones behave steadily. If that is the case, then we have a better idea of our next step.

A one to one ratio of pri ports to ip ports and you should know rather quickly if that fixes the problem.

We had an issue with the co time slots as well.We didn't have the problems that you are having, but still this might be useful. We had a 20 channel frac circuit + a 8 channel LANB card + the CKIB. We weren't using any pots lines, so we went into f24 and set the card type to none on the CKIB co's. That seemed to of take care of the time slots and cleared up the issue we were having. hope that helps a little.

Well, we finally figured this out. With the help of Dave Foster at Vertical's technical support, we found that there is a bit of a snafu in the software.

They made it possible for IP stations to receive all-call paging, but in doing so, this caused all of the IP ports to lock onto all connected IP stations. All it would take is one all-call page to cause this. The only way to break this permanent audio path is to reboot the system.

The fix was......Get this: Disable the OHVO feature at the IP stations!

I did this and went ahead and put them in page zone 0 while I was at it (just in case). As soon as I did this, the problem immediately started going away, one port at a time.

Now THAT was a tough one for sure. We have been chasing this problem for months at multiple customer sites. Thanks to everyone for the feedback.

Glad to know you got it fixed!

Why does the solution remind me of the fix for a one way audio transmission door box issue?