need some expert advice on what a telco tech called a timing issue. we have an asterisk pbx with a digium t1 card and it has been dropping a number of calls over the last two days and we're not sure what the cause is.

we had a verizon tech come on site yesterday to test the pri and he said that he was seeing a timing issue. he recommended that we restart the pbx which eliminated the issue but he only tested for a minute after the restart so i'm not convinced that there isn't some other issue. before we restarted the pbx he actually told me that he noticed a number of errors on the circuit but he wasn't sure if they were related to the timing issue.

does anyone have any recommendations as to where we should go from here?

Timing, network synchronization, clocking...whatever you want to call it, simply means that your DTE and the carrier have to agree on a common clock in order to pace out the bits that make up the voice calls. Telcos ALWAYS have more reliable clocks than your own gear, so this value is almost always set to "Network".

Look for related programming parms in your DTE and set it for "Network"; this will force the box to derive it's timing from the carrier.

I'm hoping that James will chime in here as I do not know what the Asterisk folks have chosen to rename this parameter. "Zap Omega", perhaps? They call T1 trunks "Zap" trunks, so why not? Those Asterisk people want to change everything

thanks for replying. i asked the telco to do some more intrusive testing so hopefully they will find something.

our asterisk system is a turn key box distro called switchvox. it doesn't allow you to view the inner workings of asterisk but you can set the timing provider and it is already set to network.

i'm beginning to think that there must be something wrong with the t1 card in the server....

If you have a T1, pri or otherwise that teminates in a Telco switch you must time off them. If you are having slips it can only be your problem not Telco's. If Telco had a timing problem they'd would be effecting everybody. Reseting your equipment will only work for awhile as it will eventually get out of sync again.

Pure slips will not cause errors, but massive errors can cause slips. So in answer to the Telco telling you he doesn't know if the timing issue cause the errors that answer would be no it wouldn't. So if he was seeing errors and slips it wouldn't be strictly a timing issue and Telco would need to check further.

so timing issues can only be on my side? that would really suck, i have no where to get a replacement card until after we open up tuesday morning which means that we will have more dropped calls. is there any way that the something down stream could be causing the slips?

i'm still going to try to get the provider to come out again. the tech connected his laptop to the smart jack and had 37 errors over a 24hr period. it seemed like a lot to me but i didn't think to get him to monitor a little longer while he was there.

37 errors is NOT a lot in a 24 hour period. My experience is that if there is some major problem you'll have about 20,000 errors in the first 2 seconds.

When I tested Microwave circuits we used to be allowed 1 error in 24 hours. Otherwise we had to tear the circuit down and try it again. We once did a pair of T-3 backup circuits for Bell Atlantic that ran through a repeater site across the Hudson before they got to their final destination. We could get that one down to about 5 errors in a 24 hour period and they were thrilled.

Microwave of course could take errors if a bird flew in the path of the beam. I don't think you're having that problem (the circuit isn't partially wireless, is it?), but 37 errors while not good is probably not even noticeable. Remember a T-1 is 1.544 million bits per second. That's about 92 million bits per minute, 5.5 billion bits per hour and approximately 133 billion bits per day. (I think I've got the math right.) So you've got 37 bad bits and 133 billion good ones.

You want to get it fixed, but not to agonize.

Sam

That's not enough errors, it takes massive errors to cause a circuit to slip. I would think your timing option is incorrect rather than a bad card.

No. I worked many years in a T-1 center and never once saw anything but CPE equipment cause slips. It's simple the timing source (Telco) can't be out of time. The same thing can happen CO to CO if the slave end is optioned incorrectly.

Also those errors the tech saw could have been cause by previous remote testing. When he said timing issue did he actually see slips? If not what was he basing his statement on?

first he used some software from his laptop and that is where he was seeing the slips. he then connected a t-bird to the circuit and did not see any errors. the tech was from verizon who is our local lec, but our t1 is thru att so we ended up calling att to ask if they saw any errors. while we were on hold for att, the verizon tech connected a serial cable to the smart jack and that is where he saw the errors over the 24hr period.

any ideas what could screw up the timing on the asterisk box? it was running fine for months without any issues before this then all of a suddent we start having issues.

R u sure you are set to time from the network, not. Internal, also if a t3 is hitting up u may see timing errors

positive about the network timing. pardon my ignorance but what does a t3 hitting up mean?

on another note, we've been having some bad weather here for the last 4 days, the whole time these problems have been happening. well today it was clear, until tonight of course, and the phone system hasn't gone down all day. i'm don't really think that the problem disappeared but it does seem a little strange to me. it hasn't gone this long without having an alarm for the last four days. its just weird...

What alarms are you getting? If a T3 bounces it can cause all the T1's on it to loose sync. I don't think that's your problem since there would be 28 other circuits having problems.

well, i think i spoke too soon. there is a massive thunder/lightening storm going on right now and the pbx just showed another alarm. dammit...

I looked at your other post in VoIP, D channel alarm is not a T1 alarm.

i'm not sure if the log message is related to the outages. i am getting them a lot before and after the circuit bounces but i'm not sure if they are the cause of this. they are the only errors in the log though except for the alarms of course.

isn't the d channel used for timing?

No, the D channel is used for signaling. A T1 alarm is going to be yellow red/ais or maybe oof or ses.

If you're loosing the D channel that's a whole differnt trouble.

this is the log from the last outage. let me know what you think.

24 05/25/2009 10:47 PM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
25 05/25/2009 10:47 PM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
26 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 23
27 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 22
28 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 21
29 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 20
30 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 19
31 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 18
32 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 17
33 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 16
34 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 15
35 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 14
36 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 13
37 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 12
38 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 11
39 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 10
40 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 9
41 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 8
42 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 7
43 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 6
44 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 5
45 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 4
46 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 3
47 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 2
48 05/25/2009 10:47 PM NOTICE chan_zap.c: Alarm cleared on channel 1
49 05/25/2009 10:47 PM NOTICE chan_zap.c: PRI got event: No more alarm (5) on Primary D-channel of span 1
50 05/25/2009 10:47 PM WARNING chan_zap.c: No D-channels available! Using Primary channel 24 as D-channel anyway!

50 05/25/2009 10:47 PM WARNING chan_zap.c: No D-channels available! Using Primary channel 24 as D-channel anyway!
51 05/25/2009 10:46 PM WARNING Last Message Repeated 1 times
52 05/25/2009 10:46 PM NOTICE chan_zap.c: PRI got event: Alarm (4) on Primary D-channel of span 1
53 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 23
54 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 23: Red Alarm
55 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 22
56 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 22: Red Alarm
57 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 21
58 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 21: Red Alarm
59 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 20
60 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 20: Red Alarm
61 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 19
62 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 19: Red Alarm
63 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 18
64 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 18: Red Alarm
65 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 17
66 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 17: Red Alarm
67 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 16
68 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 16: Red Alarm
69 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 15
70 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 15: Red Alarm
71 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 14
72 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 14: Red Alarm
73 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 13
74 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 13: Red Alarm
75 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 12
76 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 12: Red Alarm
77 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 11
78 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 11: Red Alarm
79 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 10
80 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 10: Red Alarm
81 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 9
82 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 9: Red Alarm
83 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 8
84 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 8: Red Alarm
85 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 7
86 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 7: Red Alarm
87 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 6
88 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 6: Red Alarm
89 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 5
90 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 5: Red Alarm
91 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 4
92 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 4: Red Alarm
93 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 3
94 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 3: Red Alarm
95 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 2
96 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 2: Red Alarm
97 05/25/2009 10:46 PM WARNING chan_zap.c: Unable to disable echo cancellation on channel 1
98 05/25/2009 10:46 PM WARNING chan_zap.c: Detected alarm on channel 1: Red Alarm
99 05/25/2009 10:45 PM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1

I don't know what the channel specific stuff means, as IP uses different terms than Telco. A red alarm at the T1 level means you're not receiveing a signal at the T1 rate. I don't know what your box says it means.

Why the IP world chooses to change terms that have been around for many years is beyond me, you'd have to ask those in that world.

me either, these logs are kind of cryptic. the thing that really sticks is that the log never says anything about timing or frame slips but the verizon tech was easily detecting them. i wish it were a bit clearer.

i'm going to run out and buy a new t1 card tomorrow morning and replace the one that i'm using. hopefully the card is the problem maker and i resolve it with a new one. if it's not then i'm back to the drawing board. thanks for all your help!

Ok, so there are a few options here.

1) The RJ48X dongle plugged into the smartjack is fubar'd and shorting out/frayed/etc. If it's plugged in unplug it and directly connect the smartjack to the T1 port. For some reason they like to get corroded and screwed up. Probably all our lovely humidity and salty air.

2) Whatever cables/connectors running between your T1 port and the smartjack is junk. If you have inline couplers, handmade cables, or other such error-prone things then replace it.

3) The T1 port on the PBX is loose/smashed/broken. Get on the phone, then walk over to the PBX and start wiggling the connector. If the calls drop with moderate agitation, then there's your problem.

4) Last is a failed hardware card. If it's an older switchvox then this is where my money is. I've been down this road before with Digium cards and they were notorious for this in their earlier production runs.

So, I'm putting my money on option #4. Given the fact that the card reports red then instantly goes green is a somewhat tell-tale sign of this kind of failure. Problems 1 and 2 usually result in an obscene about of bit errors/etc. We would be talking 6-digit error figures in a 24-hour period or more (depending on the bit error). Problem 3 is easy enough to prove with a little investigation.

Unfortunately you don't have a Sangoma card so you cant get to the real CSU/DSU error stats. If you did you could actually look up and see exactly what KIND or errors you were getting other then just "errors".

1. i'll check this out when i get to the office but the pri/smart jack were only installed less than a year ago.

2. i bought the cable from monoprice. i'll change it when i get to the office but i haven't had any issues with their cables before. of course i've only used their cables for data, perhaps voice is more sensitive?

3. i don't think this is the issue. i've wiggled the cable before and haven't seen any drops but i'll try it again when i get into the office.

4. this is probably the issue but the card is fairly new. we purchased it this january and it was the latest rev that they had back then. the switchvox was installed in august last year with a single span card but we were having all sorts of echo problems so we switched to a dual span card with hw echo cancellation. i will never use their single span cards again and wish that they support sangoma cards in switchvox but they don't. i've asked them about it before

this is part of the log from last night. the hdlc errors were actually three pages worth but i didn't want to post them all here. do you think that they are related to the drops? i don't always see them when it get an alarm but they are there often enough to make me worry about it.

100 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
101 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
102 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
103 05/26/2009 5:41 AM NOTICE Last Message Repeated 3 times
104 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
105 05/26/2009 5:41 AM NOTICE Last Message Repeated 1 times
106 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
107 05/26/2009 5:41 AM NOTICE Last Message Repeated 1 times
108 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
109 05/26/2009 5:41 AM NOTICE Last Message Repeated 1 times
110 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
111 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
112 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
113 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
114 05/26/2009 5:41 AM NOTICE Last Message Repeated 1 times
115 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
116 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
117 05/26/2009 5:41 AM NOTICE Last Message Repeated 2 times
118 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
119 05/26/2009 5:41 AM NOTICE Last Message Repeated 3 times
120 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
121 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
122 05/26/2009 5:41 AM NOTICE Last Message Repeated 1 times
123 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
124 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
125 05/26/2009 5:41 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
126 05/26/2009 1:27 AM NOTICE Last Message Repeated 1 times
127 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): Error releasing mutex: Operation not permitted
128 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): mutex '&q->lock' freed more times than we've locked!
129 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): Error releasing mutex: Operation not permitted
130 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): mutex '&q->lock' freed more times than we've locked!
131 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): Error releasing mutex: Operation not permitted
132 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): mutex '&q->lock' freed more times than we've locked!
133 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): Error releasing mutex: Operation not permitted
134 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): mutex '&q->lock' freed more times than we've locked!
135 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): Error releasing mutex: Operation not permitted
136 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): mutex '&q->lock' freed more times than we've locked!
137 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): Error releasing mutex: Operation not permitted
138 05/26/2009 12:02 AM ERROR ../include/asterisk/lock.h: app_queue.c line 4506 (manager_queue_reload): mutex '&q->lock' freed more times than we've locked!

we're having good weather and the issue seems to have magically disappeared again. i'm not sure if it's a coincidence or not but i'm not really sure that we need a new card and i can almost guarantee that the issues will start up again as soon as the weather worsens.

Well, there is a lot of bad copper in Tampa. It's not uncommon to see a cable get 500-1000 feet longer when it rains.

With the D-Channel flapping like that it would explain all the dropped calls as well.

Looks like you've got a bunch of deadlocks too (All the queue messages). It's not related to your dropping calls but I figured I'd mention it as well.

I've got a guy who is doing some work for us that has a bit of experience with SwitchVox. You can try giving him at call at 863-393-9336. His name is Kevin.

we're having issues again and it's just thundering.

so bad copper could cause hdlc errors on the d-channel which would make the circuit bounce?

Bad copper can make the circuit do anything from noisy background to total service outage.

BTW, I posted the wrong number. That's my office line. You are more then welcome to call it but I'm usually comatose until around 10:30 in the morning. One of the side-effects of being the 3rd shift maintenance crew.

Kevin's number is 863-393-9339.

digium tested all the equipment last night and everything tested fine though we're still having a problem.

this has to be a telco issue then and it's like pulling teeth to get them to come out and actually find the root cause of this. you would think that they would acknowledge that its their problem since it only happens when the weather is bad...

Than you have more than just a slipping problem. Sure would like to see some T1 test results, but saw in your previous log you don't have that ability.

Does the system have an internal or external CSU? If it is external, has that been checked? (I quickly scanned the remarks, and may have missed the reference).

Some smart jacks do have the ability to log errors & alarms. If such a device is being used, Verizon can read the log.

I'm betting all those red alarms that show channel specific are red alarms on the T1. If so that means you are loosing signal. Which should have shown at the smart jack when the tech put his laptop on it. I've never seen any piece of equipment that show red on a channel, but I've never messed with VoIP stuff.

it's an internal csu.

although asterisk has the capability to use voip, we're not using it for outbound calling. the pc has a t1 card that the interfaces with the smart jack similar to most digital phone systems from what i understand.

here is a snippet of the latest log. today has been an absolute nightmare as far as the drops go.

100 05/27/2009 10:35 AM NOTICE chan_zap.c: PRI got event: No more alarm (5) on Primary D-channel of span 2
101 05/27/2009 10:35 AM WARNING chan_zap.c: No D-channels available! Using Primary channel 48 as D-channel anyway!
102 05/27/2009 10:35 AM NOTICE chan_zap.c: PRI got event: Alarm (4) on Primary D-channel of span 2
103 05/27/2009 10:35 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 47
104 05/27/2009 10:35 AM WARNING chan_zap.c: Detected alarm on channel 47: Red Alarm
105 05/27/2009 10:35 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 46
106 05/27/2009 10:35 AM WARNING chan_zap.c: Detected alarm on channel 46: Red Alarm

Maybe I'm looking at this wrong (and maybe it could be just an "Asterisk" thing) but the error logs clearly identify a "Primary D-Channel" of "span 1", and also a "Primary D-Channel" of "span 2". I also notice 46 "B-channels" throughout the error logs, which tells me that your system is configured for at least two PRIs.

Is this correct?

Were you receiving these "No D-channel Available" warnings before you "restarted" the system?

sorry for the confusion. we only have one pri but the t1 card that we use actually has two ports. last night after digium tested the card we decided to plug the t1 into the second span to see if it would make a difference. it obviously hasn't so we will be moving it back soon.

regarding your question about d-channel available warnings. we were still getting them before we rebooted the server as well.

If possible, can you post any new errors after moving the T1 back to the original port?

sure will. we haven't had a problem for a couple of hours so i've been holding off on making the switch but i will do it after the office closes it and will post the log after the next red alarm.

att is currently running intrusive tests on the circuit which is not allowing me to get an accurate log. they want to do a class a cable inspection but they said that they will charge us $1500 every four hours if they do not find any issues. is this normal?

also, should they be able to detect every red alarm that we see if they are monitoring the circuit?

Not know which at&t local or ld hard to say. They won't see alarm history in the at&t stuff but they will see a history of oof's when the circuit went down. Also depends what equipment they are looking at. I'm talking from the LD side. The charging sounds like you dealing with at&t local service.

verizon is actually our local telco but our t1 is thru att.

Well at&t used to terminate in 4E's for pri, not sure about now. They should have places to monitor and if you're getting red alarm on the T1, which I don't know if you are or not, they will have an alarm history of yellow at the 4E, unless it's failing both ways. Not knowing the circuit layout or what's on it it's really hard to say. If it's failing in the local access and they can't see alarm history they would have no information. Hopefully a stress test will find it if it's in the access.

well i just heard from att and they found a problem. they're not telling me what it is as verizon has not released it to them but they schedule to dispatch someone tomorrow morning. i guess we'll have to wait and see. i just hope they don't start blaming the pbx again.

kumba, i tried to call kevin's number a couple of times and got tom's voicemail.

verizon tech is on site now but here is a log after switching ports.


500 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 16: Red Alarm
501 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 15
502 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 15: Red Alarm
503 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 14
504 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 14: Red Alarm
505 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 13
506 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 13: Red Alarm
507 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 12
508 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 12: Red Alarm
509 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 11
510 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 11: Red Alarm
511 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 10
512 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 10: Red Alarm
513 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 9
514 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 9: Red Alarm
515 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 8
516 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 8: Red Alarm
517 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 7
518 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 7: Red Alarm
519 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 6
520 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 6: Red Alarm
521 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 4
522 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 4: Red Alarm
523 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 3
524 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 3: Red Alarm
525 05/28/2009 8:20 AM WARNING chan_zap.c: Unable to disable echo cancellation on channel 1
526 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 1: Red Alarm
527 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 5: Red Alarm
528 05/28/2009 8:20 AM WARNING chan_zap.c: Detected alarm on channel 2: Red Alarm
529 05/28/2009 8:20 AM WARNING chan_zap.c: No D-channels available! Using Primary channel 24 as D-channel anyway!
530 05/28/2009 8:20 AM NOTICE chan_zap.c: PRI got event: Alarm (4) on Primary D-channel of span 1
531 05/28/2009 8:20 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
532 05/28/2009 8:20 AM NOTICE Last Message Repeated 1 times
533 05/28/2009 8:20 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
534 05/28/2009 8:18 AM WARNING ast_expr2.y: non-numeric argument
535 05/28/2009 8:16 AM WARNING Last Message Repeated 12 times
536 05/28/2009 8:12 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
537 05/28/2009 8:12 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
538 05/28/2009 8:09 AM NOTICE chan_zap.c: PRI got event: HDLC Abort (6) on Primary D-channel of span 1
539 05/28/2009 8:09 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
540 05/28/2009 8:09 AM NOTICE Last Message Repeated 1 times
541 05/28/2009 8:08 AM WARNING ast_expr2.y: non-numeric argument
542 05/28/2009 8:04 AM WARNING Last Message Repeated 13 times
543 05/28/2009 8:01 AM NOTICE chan_zap.c: PRI got event: HDLC Bad FCS (8) on Primary D-channel of span 1
544 05/28/2009 8:00 AM NOTICE chan_zap.c: PRI got event: No more alarm (5) on Primary D-channel of span 1

I'm still bugged by that T3 comment. I know most of you are thinking about facilities, but he could have been referring to the LAPB T3 timer. Yeah...remember that from the Frame Relay days? Well it's used in all kinds of stuff from VoDSL to NFAS PRI circuits.

It's a "watchdog timer"--as I like to call it--which says, "how long do I wait to hear from the other end before I claim that the link is down and bounce this circuit?"

Kinda makes sense here, although it might be a red herring. It would be neat to know how exactly it is that the carrier is delivering this PRI.

this is the latest update from our att service manager. vzn tech is still on site though so there may be more issues.

FOUND BAD ISOLATE PAIR(F2) 3090-171(XBOX) 151(TERMINAL) HA S METALLIC FAULTS,MOVED IT TO 3090-179,CHECKED OTHER PAIR

Carriers are a pain in the you-know-what when it comes to these types of troubles. Nobody thinks anymore. You see a failure, you report it to the carrier. 75% of the time a tech doesn't touch it with AT&T long distance, it goes to an automated system that auto-tests your circuit. What does it do? It loops up the smartjack, tests to it, then loops up the CSU, and tests to it. Of course it finds nothing because hey, test ok, the alarms have already cleared so therefore it MUST be your fault (which is BS).

Unfortunately a lot of the techs aren't better. They'll take a window, they'll test, and that window is just the gospel of anything and everything with your circuit. A lot of times they don't even stop and LOOK at what they are seeing.

A lot of smartjacks have performance monitors built in to them (sometimes referred to as a genius jack). I wonder if before intrusively jumping into the circuit and testing it somebody has actually looked at the alarm history both at the smartjack as well as the test points in AT&T's network to see who's seeing what from where. I never understood how people can just jump in and hit a button to have a little computer tell them what the problem is without understanding to look at the orientation of the errors/failure!

If I seem a bit harsh it's because I used to work in a remote test center for a carrier and this became a serious pet peeve for me as I watched them and other carriers dumb down their techs to the point that they became test system monkeys. They push the magic button, it does the work for them, and they spit out the result without any thought. Works great for most cut-and-dry failures, but as soon as you get something intermittent it becomes a problem.

The red alarm you're seeing on your system indicates a loss of framing. As a result, your system would generate a yellow alarm to transmit towards the carrier. If the path to the carrier was good, that signal would get to them and they could see it on their alarm history. If it was not, they would see history of AIS.

The point is that they would see something. I would bet money that nobody has even looked at the history, and now doing so would be useless as the data would be corrupted with intrusive testing.

If AT&T comes back after Verizon changes the cable pair as a problem with Verizon and then closes their ticket, and the trouble comes back, tell them you want them to monitor the PMs for a few days. When the trouble occurs again, call them up, and ask them what the PMs show. At least then you can get a handle on the orientation of the fault. The same can be done with Verizon if AT&T asks so they can pull the PMs off the smartjack (which if it's a 'genius jack' they can do remotely as it will store the PM info...in fact, AT&T can get it too if they know how which saves a lot of effort). That way the next failure can go like this:

"My T1 just failed again, I have a red alarm on my CSU. That means I lost framing." --You

"Red alarm history on the smartjack too, must be failing before here." --Verizon

"AIS history on the first port...must be an issue between this port and the smartjack, let's start changing things out to see if it helps." --Verizon


Of course, it's never that easy, but it beats the hell out of chasing after a ghost with random T1 testing. If an intermittent issue not presenting itself on a long intrusive test then it's a crap shoot to catch. It's much easier (and more logical) to look at the errors/alarms you're seeing and start making educated guesses.

Sorry for the long rant.

Kumba, can you please give me kevin's number again. the one that i have seems to be someone else's.

863-393-9339

He probably hasn't recorded over the other guy's voicemail message. I'll have to yell at him about that.