Hello everyone! What a wonderful resource I seem to have found today. I'm hoping you can all point me in the right direction to troubleshoot this problem I have been experiencing.

It's long and boring so here's your chance to bail.

Ok so a little info to start.

I work for a CLEC that provides T1 voice and data service to the NJ,DE, and PA areas.

This customer has two T1s with us. One is used for data and the other for voice.

Both T1s come into a Tellabs Titan 5500 DACS over different DS3 facilities and cards. One T1 is then cross connected to a Cisco 7500 for data. The other goes to a 5ESS for voice.

At the customer side, the T1s go into a Carrier Access Adit 600. The data is routed connected to the router card in slot 6. Voice is terminated to a 66 block through FXS cards.

One day the customer calls in to us and complains that she cannot send long/large faxes. We do some homework and to cut right to the chase, find that the voice T1 is taking RX slips but nothing else. The data T1 in the same router is clean with no slips.

We go intrusive on the line and test to a tberd and see no errors at all. Not even slips.

We open tickets with the local carrier (Verizon) and have the line tested outside of normal business hours. They have the same results we do.

If the circuit is broken out to test, the errors stop. When it gets put back together, the errors return. Again, ONLY rx slips.

So, we dispatch out to the customer and replace hardware. Same problem. We then decide to move service from one t1 to another and essentially swap data and voice from one t1 to the other.

The errors followed the service meaning the t1 that was perfectly clean before with data now has rx slips on it with voice. The other circuit that was taking rx slip errors with voice is now 100% clean with data on it.

This is where I think I will stop. I could add more but I would like to see what everyone here may have to add to this.

I have a feeling that it is something simple and easy to many of you and it's probably staring us in the face here but it's being overlooked.

Thanks in advance for any assistance.

Fred

I'm not totally familiar with the Titan. We're using a 532-LS, which is similar. Clocking can be a real PITA in a network.

The first thing I would check would be the timing source on the DS-3's. Check the muxes to see if they are both the same.

From there, I would check the timing in the Titan on the DS-1's to make sure both circuits are provisioned the same. This may need to be checked on the HSPM cards as well.

Two things. By RX slips I assume you mean receive, slips are not directional. Pure slips can only be caused by end user equipment. I'll make it 3 things, if you test good to a T-Berd, I again assume you are removing the customer equipment and therefore removing the source of the slips. You would have to monitor the live circuit using the circuit timing to see the slips.

If you have a T-Berd 224, you can monitor the circuit in both directions.

The Titan 5500 is similar to the 532L, but it is a wideband DCS (3->1). Default setting for timing is "Thru", meaning the Titan will not provide a clock signal (it is transparent).

If the circuit facing the customer is set up as a CST, you can check the setting by using the "RTRV-T1::xxx-xx:(ctag);" command and look for the timing datafield. You could also check the connected side as well (I think DGRs also have the timing datafield).

If the problem follows the device, as opposed to following the T1 it might indicate a problem with a T1 card in the CPE (hardware or software) or even a marginal cable between the CSU (if one is there) and the PBX. Also have the CPE programming checked to make sure that it is timing off of the T1 span as opposed to its own internal clock (aka free-run).

An overnight or over the weekend monitor with the T-Berd might be helpful.

I will check the settings in the DACS when I get in to work in the AM. Thanks for the command.

As for the CPE and the T1 Circuits, please note the following from my original post:

So, we dispatch out to the customer and replace hardware. Same problem. We then decide to move service from one t1 to another and essentially swap data and voice from one t1 to the other.

The errors followed the service meaning the t1 that was perfectly clean before with data now has rx slips on it with voice. The other circuit that was taking rx slip errors with voice is now 100% clean with data on it.

This is what bothers me. The slips only appear on the circuit that is carrying "voice" traffic or terminates on the 5ESS. There's an overwhelming majority of people within my company that seem to believe that the 5E cannot be the problem or it would affect everyone using us for voice. I can sorta buy that but how can they explain movin voice from one T1 to the other T1 and having the errors follow the voice traffic?

Thanks so far for all of the help everyone! Keep it coming. I have a scheduled vendor meet with Verizon in the AM even though they have been testing both circuits clean.

Fred

The errors would follow the T1. You're taking the working T1 and putting it into the CPE that is causing the slips. Pure slips can only be caused by CPE equipment, that CPE can be a DACS, or whatever it terminates in. Once again take your TBerd, and time it off the good circuit while monitoring you will see the slips. The string command for checking for slips is fine, most carriers (Verizon) don't know how to test for slips, they don't care about slips, it's not their problem. If Verizon were causing slips than many thousands of T1's would be slipping. Make sense? The supplier of the clock can't slip.

Thats right, since the problem followed, it must be on the cpe side, or a wierd compatibilty issue between the cpe and CO equipment. Could also be as simple as the cable between the DTI card and the smart jack.

My guess is that you will find the CPE is not set to use the T1 as a timing source.

It is probably set to use an internal driver and unless that internal driver has stratum 0 or 1 accuracy, timing slips will be seen if you monitor the T1 long enough. It might be slow bit slipping, or it could be a full blown free-run condition.

Just to note, we verified that the customer router is configured to time off of the loop.

The 5E (on the opposite side) is a master timing source and is connected to a bits clock.

I did the command you suggested and it appears that both T1s are timing through. So the dacs and T1s are or appear to be configured correctly. The router at the customer prem is set to loop timing. The 5E is a master clock source using a bits clock.

Now what? Leaving in 30 minutes for a vendor meet.

M 8575 COMPLD
"0072-08:CST:PMAID=PM3-0072,TACC=0,IDLECDE=AIS,OOSCDE=AIS,FMT=ESF,GOS=99,ALM=ALW,ALMPF=99,LPRSP=YES,,,
FENDPMTYPE=ANSI403,DS1ADDR=C,CSUADDR=B,TMG=THRU,FLTRC=NONE:PST=IS-NR"
;
RTRV-T1::95-11:8576;


M 8576 COMPLD
"0095-11:CST:PMAID=PM3-0095,TACC=0,IDLECDE=AIS,OOSCDE=AIS,FMT=ESF,GOS=99
,ALM=ALW,ALMPF=99,LPRSP=YES,,,
FENDPMTYPE=ANSI403,DS1ADDR=C,CSUADDR=B,TMG=THRU,FLTRC=NONE:PST=IS-NR"

The Titan appears to be provisioned correctly (except ALMPF=99 which will never generate visual alarms should the T1 take a dive).

In this instance, assuming that the Titan and 5E are using the using the same bits clock, you could try to use the Titan to provide clocking to the CPE. It is possible that the CPE doesn't like the timing bits from the 5E.

This may be seem like a silly question but are you lookin’ for and seeing the slips on the voice T1 at the DS0 level or DS1 level?

Man Dex… I thought you had it with this one! I would have bet money on a timing source mismatch.

I am at the customer location now. verizon came and went. t1 tested clean on all patterns head to head tberd to tberd. no slips.

I believe we are doing t1 level testing and monitoring.

Should I be doing something else?

Can this be a bad extension? Can there be bad options somewhere causing this?

thanks again for the help!!!

posted from my treo.

fred

Just for giggles, on the CST facing the customer, change the timing from through to the other option (INT?).

Let's see if the CPE likes the timing bit stream from the Titan.

If no change, I'm stumped. :confused:

Thanks for the post Bryan

I also thought I had nailed it. :scratch:

What is the voice T1 connected to when it leaves the Titan on the C.O. side? Does it go directly into the 5ESS as digital or does it go to a channel bank where the 24 circuits are changed to analog and wired to analog OEs. If there is another channel bank there could be a timing issue there.
Also, what does the T1 ride on going from the Titan to the customer? Is it on copper or does it ride in something like a Litespan? That could be another place for a timing problem.
I’m like everyone else. Timing has to be the issue. I know that fax machines being served by a SLC96 will have problems is the SLC remote terminal is not set up to time from the C.O. end.

GOOD question!!!


I read through this entire post one more time to see if I could rattle something loose in my memory. (Man I wish some of you would have typed slower!!! I cain’t read that fast! )

One thing I (or us maybe) may be loosing you on is… Was the not being able to send long/large faxes the root problem here (problem with that one trunk) OR was that just the symptom that caused you to see problems on the entire T1 delivering the voice trunks?

Ok I just got back from the customer site.

Here is some added info for yas.

1) This was initially a fax complaint which pointed us to the T1 taking slips in the customer router.

2) The connection from the Titan DACS to the 5E switch is via direct cable. There is no mux in the middle. Sorry. Would have been nice and possibly the issue.

3) We tried to change the timing on the T1 in the titan to internal from through but it didn't change anything. Still got the slips.

4) We put a duplicate router right at the dmarc and still got slips. Rules out house wiring.

5) With the router plugged in at the dmarc and mapped all the way to the switch, we get slips. If my tech at the office changes the cross connects for the T1 in the Titan and maps it over to his tberd, the errors STOP. NO ERRORS!

I am being told to have Verizon verify the signaling and coding on the T1 at all points. He seems to believe that there is a hop in the circuit that is not optioned correctly. If that was the case, why is it that the T1 runs perfectly fine with data on it?

Remember:
T1 1 had voice and T1 2 had data.
T1 1 had the slips. T1 2 was clean.

We swapped the services so we ended up with:
T1 1 has data and T1 2 has voice.
T1 1 is clean and T1 2 is taking slip errors.

I am going out of my mind here!

Thanks again for the help. Hope these answers can assist.

2) The connection from the Titan DACS to the 5E switch is via direct cable. There is no mux in the middle. Sorry. Would have been nice and possibly the issue.

Still could be a bad trunk side port at the switch or the DSX-1 jack it's tied to.

From what I understand, it goes to the switch via DS3 Coax.

We also thought it might be a bad card or cable so we had the switch facility moved to a new one and there are still slips.

Ahhhh crap :scratch: LOL

5) With the router plugged in at the dmarc and mapped all the way to the switch, we get slips. If my tech at the office changes the cross connects for the T1 in the Titan and maps it over to his tberd, the errors STOP. NO ERRORS!

.... <sigh> this STILL points me to the problem being bewteen the Titian DCS and the 5E. Was the office tech changing the circuit to go to and thru his T-Berd to the switch or just ruuning off it? (not going to the switch at all during that testing? :confused:

I figured I'd chime in here. I work at the same company and have been involved with the problem. Our switch technician was remapping the circuit /TO/ the Tberd, not through it. As in, with the 5E completely out of the loop, all errors disappear. Once the 5E is brought back into the mix, the timing errors resurface.

not going to the switch at all. cross connect was changed to a test port on the titan which went to the tberd.

posted from my treo.
fred

Have you tried moving this circuit to a different switch mod?

Are we talking about a VOIP circuit? Why the router? If we are there lies the problem.

It looks to me like an issue between the Titan and the 5E. Can you put a router in the CO on the customer NPC coming out of the Titan, with a T-BERD at the DSX-1 setup to monitor both directions?

You will not see the slips taking the circuit down and running to a TBerd, that's all there is too it, you HAVE to monitor the circuit and time the TBerd off a known good circuit to see slips. Just exactly where are you seeing the slips? 5E? DACS? Customer location? If you are seeing them at the 5E you will also see them at the DACS, assuming this goes cust, dacs to 5e. Slips are not directional, they are on both sides equally.

I am being told that we cannot see slips in the 5E and I mean as it's not an available PM in the 5E. I don't know much about the 5E so I cannot comment further. Same thing goes for the dacs. I am told that slips don't show up in the PMs on the dacs. Really helpfull, I know.

We are seeing the slips at the customer location in the integrated voice and data unit that is out there. It is NOT voice over ip. This unit acts as a miniature dacs and allows us to map x channels for voice to a 66block and x channels for data to a router card.

And Kyle, regarding your question. We did something like that. We took a duplicate of the device that is at the customer prem and set it up in our lab here. We then took the T1 at the dacs level and remapped it to the device in the lab. It showed no slips. So in this case, it goes 5E to Titan and then Titan to device in lab. No errors. I don't recall if a tberd was in the middle watching both directions for this test though but since it showed no errors we simply moved on.

Again, I can't thank everyone enough for trying to help. You guys are a great bunch.

What is the Make/Model of this unit you are talking about?

~r

It's a Carrier Access Adit 600. (I'm certainly no big fan of them) It has (2) 8 port FXS cards in it and a router card in slot 6.

Just for reference, I swapped that out with an Adtran 912 (Love adtrans, hate the tech support wait.) and still got the controlled slips.

depending on what type of OE this is built on in a 5e will determine what errors you can see. If it is on an IFAC, you will most likely not see the slips. If it is built on a NEN you will be able to see coding violations, errored seconds, severely errored seconds, severely errored frame seconds, unavailable seconds, and controlled slips

I am being told it is a NEN and we are seeing slips on the 5E interface. About 1 per 15 minute interval so I am told. We are seeing about 1 or 2 every minute and an average of 34 every 15 minutes.

to be honest it is very very rare for me to see the trouble originating from the switch, it is almost always something out towards the customer. Especially if you try to roll this to a new SM and the trouble follows. I have had similier troubles in the past, for every errored second I had, I had an identical slip, looked like a timing issue.It would often turn out to be a defective t3 card. switch it over to protect and all cleared..

1 quick way to determine if the slips are being caused by the 5E would be to compare the output of the suspect switch T1 with the output of a known good switch T1.

If the suspect and the chosen good T1 spans are both CSTs you would have to map the circuits to TAPS and connect to the monitor jacks on the DSX panels.

Sorry meant to edit not quote.

A couple of other thoughts. I’m grasping for straws. Is the 5E a host or is it a remote? If it’s a remote then there are other network elements that come in to play. Can the C.O. tech turn on the switch cutoff call report to see if it shows anything? The Plant 24 Report gives the number of switch cutoffs for the whole switch. Turning on the Switch Cutoff prints out a detail of every call that is classified as a switch cutoff. There used to be another report that you could turn on called the transient cutoff report. That could point something. Maybe there are others having a problem that you’re not aware of. I really want to hear what happens on this.

Well boys and girls...and bf6b5yr, we just got another complaint of fax troubles and it appears as though the customer has two T1s one of voice and one of data and guess what.... there are slips on the voice T1!!! Not sure if I should be happy or sad but we are doing some work to see if the customers have anything further in common.

I would start looking at this at a t3 leavel

I'm moving this to the T-1 category. Majestic, just remember one thing, the timing source can't cause slips, that should help you trouble shoot your problem. I'm not real good at T3 so I'll let anthony deal with that part. You have a timing option set wrong somewhere, that is the only thing that will cause pure slips.

Can you find out what the two failing T1s have in common. Anthony has a good suggestion. Find out if they ride on the same T3 and try switching the circuits to the protection card.

Bill’s right, ONE GOOD timing source can’t cause slips.

BUT… One BAD (or going bad) timing source can cause slips.

Also, TWO or more good timing source trying to control the timing of a circuit can cause slips.

Hopefully that’s already understood… I’m pretty much out of ideas on this one… :rolleyes: Well, good ideas anyway.

Let me clarify that just a bit Bryan. I was talking source as in Verizon, AT&T ect. As everything from the carrier would be bad. CPE timing sources can go bad as you pointed out.

Also, TWO or more good timing source trying to control the timing of a circuit can cause slips.

I am of the belief that this is what is causing this.

I just proactively found 3 more customers that are all taking slips on the t1 that serves voice. They are all multiple T1 customers.

They are all in different latas and ride over different DS3 facilities.

Again, the only common thing is that they are connected to the 5E for voice and our ciscos for data. The Cisco's are 7500s that are set to time off of the loop. The 5E is using a bits clock.

have you gone as far back as the oc3 (or 12 or 48, or whatever you are using)

Different circuits, riding different rings, may still ride the same oc3. I really do not think the problem will be in the 5e

I hear ya Anthony even though everything seems to point to the 5E because when it's not in the picture, neither are the slips. As many of you might imagine, I'm quite frustrated with all of this.

I am working with my supervisor today and seeing how we can escalate this to have the OC circuit facilities checked.

I wouldn't think it would be larger than the T-1 level or you'd be having a lot more problems, especially on data. If the only common thing is the 5E that's where I'd look. Anthony, when a 5E is used for a tandem switch it's slaved off the master clock isn't it? Don't know how you could have two 5E's in tandem with both suppling clock. I'm sure it would be at the T-1 level in the DTU (maybe not the right term for the T-1 cards)

Was this problem ever resolved? Lots of good help has been given!!

Not as of yet. We are still working on this. Sadly I don't see much progress.

Well boys and girls it is with great pride that I come to you today with great news!

Yesterday we were finally able to pinpoint some crucial differences and similarities that gave us a way to stop the slips.

1) All of the customers have at least 2 T1s or more.
2) All of the customers had at least 1 T1 configured for voice (went to a 5ESS) and 1 configured for data (went to a Cisco 7500).
3) All of the customers had a Carrier Access ADIT 600 router and both (or more) T1s went into it.
4) All of the Carrier Access ADIT 600s were set to time off of A:1 which is the first T1 slot.

Now here is where things were different...

Some of the customers had the voice going to T1 1 and the data going to T1 2. Some were the other way around.

Those that had voice on T1 1 and slot 1 was the timing source were fine on all ports.

Those that had data on T1 1 and slot 1 was the timing source took slips on the "voice" T1 which would be in a different slot.

Changing the clock source for the router to use the VOICE T1 (regardless of slot) stopped the slips from occurring.

Now the big question is, why would this be an issue? We time off of the loop for either voice or data. The real difference is that the 5E has a BITS clock and is set to be a master and the Cisco 7500s are set to recovered/loop timing.

Thanks for everyone's help with all of this. I'm glad that I can post a solution to the problem. (Even though I still would like to know the difference of timing on the voice T1 instead of the Data T1)

Fred

Since slot 1 provides your timing to all the other slots, the only thing I can see is possibly when you set it to voice it slaves, but when you set slot 1 to data it provides clock. Not knowing this piece of equipment, that is just a guess on my part, but seems logical.

I have to say this one more time. Pure slips can only be caused by an end user, not a supplier. That of course assumes a supplier on one end. Where I was getting a little confused is I kept reading this like your 5E was an end user from a supplier, which in such a case the same would be true, because your 5E would have to time off the networks supply. There is only one master clock source and everyone has to time off it.

Majestic, thank you for letting us know what resolved your problem.
Bill

GOOD NEWS Majestic,

I don’t think it’s a difference of taking the timing off the voice or data. It’s the difference of the CLOCK SOURCE between the voice and data.

The way ya’ll are set up you voice T1 has a good stratum 1 (maybe 2) clock source directly connected. BUT the way ya’ll have your data router set up, it’s only able to pass a “second hand” timing source.