Let me start by saying, if this post should be in a different category, mods- feel free to move it. I didn't see anywhere it really belonged, and it's borderline within the scope of these forums anyway...

As many of you know, I do the maintenance for a local bar. This is the site where the trouble is.

About a month or so ago, when we had our internet service from Consolidated un-suspended, we started having all kinds of issues with the DSL. Prior to the human-malware shutting down the world, this connection was relatively trouble free for the better part of a year. When the bar was forced to close, we put the internet on suspension (They shut it down, charge you a little under half price, and with a phone call it's back on in a day... in theory)

Like I said, before the plague hit, this thing worked almost flawlessly for a year.

After we reopened, the thing is down almost every day.

When I say "down" this may be an important clue... The two circuits stay synced up. The lights on the modem for DSL1 and DSL2 stay orange (meaning vdsl, as green means adsl) The "internet" light goes out and we have zero connectivity to the public internet. The LAN seems to stay up, but whoopee do...

The details:
40/10 Bonded VDSL, 2 "dry loops" ; I put dry in quotes because Consolidated actually puts battery on the pairs and a repeating message telling you the circuit phone number (To help prevent people cutting wires they think are dead, and damn handy too)
Nothing on our side had changed; same network topology, same router/modem, same connected devices, same everything. It was like a 4 month time capsule.

First service call:
Tech finds a couple bad pins on a 66 block and a bad "A" side, since it was basically operating on 1/2 a pair. When he left, pairs were good, system stayed up a day and a half... then kaput
I go to the cellar and check everything. Decide I don't like the looks of some of the pins on the blocks and swap us over to some new pairs in the house cable. Line sounds clean, modem syncs up, works for 2 days, kaput.

Second service call:
This tech was about as useless as they come. Because it was working correctly when he showed up, he wouldn't do anything. Except talk, and talk is cheap. He explained it seems like a loss of rout, not sync (duh) but because it was working, NTF.
If stupid customer service didn't have me reboot it as the first part of the call, it would still be out.... Then again, I can't run a bar with no internet, so either way I HAVE TO REBOOT THE GODFORSAKEN THING.

Third service call:
This tech is a cool dude... He is quite done with the company getting angry calls from me about my modem being down. He goes to the basement, tests the pairs in the terminal back to the CO with his sidekick, they look good. Goes to the bar office, checks the pairs at the jack. They seem perfect. He takes the modem, tosses it in the trash and installs a new one, new phone cord and all. Goes back to the CO, puts us on a different port on the OCCAM blade DSLAM, comes back out, runs some tests, declares us all good and leaves. Quite thorough, and more so than I've come to expect from CCI employees. 2 days later it's down again. A day after that it's down again.... and so on, and so on...

Every time its the same condition. Modem appears to be still synced up, but no routing. DSL lights are on solid, internet light is out. Reboot and it all comes back to life till it does it again.

Aside from bailing and going to the competition, anyone have any clues what the problem could be?

Tech #2 was convinced I have "too much on the network". I call BS, but sure... There's a tablet for a POS terminal, a Roku for the TV, and a laptop in the DJ booth. We also have a camera system which he was CONVINCED was the problem, so I pulled the ethernet cord out of it and the thing continues to crash. So, sorry try again. It uses about 1.5 Mbps upload and about nothing for download... so not even close to taxing the connection. And for the previous year it was connected and never had an issue.

I've tried isolating everything on the network, and nothing seems to be a cause. Any ides would be greatly appreciated.
Thanks

Chris

Not knowing how CCI does their bonded dsl, I can take a few guesses... the first would be that they're using multilink PPPoE (or whatever they want to call it, PPP over the dsl line) and that the ppp server on the CCI end is crashing, puking, getting enough bad packets to go walkies. Not everybody does PPP, though, some actually stuff IP directly on the line, so...

A thing to try... can you get internet with only 1 pair connected? (it should, if it's set up right) If you can, does it stay up more than a day? does pulling 1 pair and replacing it make it re-sync and be happy? This all sounds like a networking thing, not a physical hardware thing. You're getting dsl sync to the dslam, you're just not getting any IP traffic. Do you know anybody else who uses CCI dsl out of the same CO? are they having similar problems? If they are, that points more toward a problem in the CO itself... losing the connection to the radius server, losing the connection to the outside world from the CO...

Just a few guesses, CCI's more a phone company than a network company, so getting the right person to troubleshoot may take... a lot of work. if it's something in the CO, it SHOULD be sending alerts to their NOC, but they may get so many low-level alerts that they just ignore them... "oh, it alerted, but an hour later when I checked it, it was working" (because the customer rebooted the modem, which forced a sync and a new request for an IP)

Good luck!

^^^This^^^

-Hal

The only thing remotely close to this scenario I've seen is when one of my radio station clients thought they could speed up their computer by turning on the built-in WiFi connection on their desktop PC, while it was still connected via Ethernet. A few hours later, they complained they lost the Internet, rebooted the DSL modem, got service back, speed got slowly worse until it was dead again, etc. Told them to disable the WiFi in the desktop PCs, which fixed the problem. I explained to them in broadcasting terms they understood, that they had created a feedback loop by doing what they did.

We've run into almost this same problem in the past with some of our DSL customers. The DSL sync would never drop, but the Internet light would. Came to find out our IP pool was completely saturated and we had no IP's to spare. I imagine with Covid and everyone working from home, that there have been all sorts of new installs and that this is the issue. Once we added IP's to the pool, the problem went away. Sounds to me that is what CCI needs to be looking at.

What happens when you connect a laptop straight to the modem?

Chris
DSL is very old technology. Move them to the local cable company and put a end to these
problems.
Ken

Yet it's all I can get, except for pricey and slower satellite. I'm in the country, but close enough to town I can get 12meg, which after many years of 1.5 seems great.

this could easily explain it... DHCP gets a lease for an IP, it lasts until it hits it's timeout... 24 hours is a common DHCP timeout...

That's something my clients haven't experienced...they all have static IP's.

One of the offices now has fiber, and we're switching them over soon.

I also have glass at home. Other than service loss during a week-long power outage (until they got generators deployed to the cabinet across the street), it's been very good.

Thank you all for the input. Couple of things to note.

We have talked about going to cable, it’s on the table, but the fees associated with switching (early termination fees, installation fees, etc) are prohibitive. Not impossible, but it makes it a bitter pill to swallow. The owner and I have discussed it, if we can get CCI to waive the ETF somehow, we may make the switch.

The latest thing I tried, before reading these posts, was to connect only one PC to the gateway via Ethernet. I turned the WiFi radios off, so the only connection would be this one PC that has very low network usage, and after a day or so, the same thing happened, DSL sync is up, internet is out.

At this particular moment, I have left it in that state so the tech coming tomorrow can see it.

If I had my way, we’d switch to cable, I’d port the phone to Voip.ms, and call it a day.

I will ask the tech about the ip pool situation and see if exhaustion is the problem. This did all start with COVID.

I will try the one pair test if it continues to be a problem after tomorrow. I don’t want it to resync before the tech gets here.

I will post an update once I have more.

I can't see it as an exhaustion of your IP pool, rather an exhaustion of your service providers capacity.

I would get an attorney to draft a letter giving them a time limit to get this resolved or the situation will be considered a breach of contract and no ETF will be due.

-Hal

I reached out to the first tech that came out. He seems like a really decent guy, who cares about doing a good job and wants to do right by people. We've had a little bit of texting dialog, and he reached out to me this afternoon.

He had the CO tech look at the circuit and said 'It looked terrible'. He said he'd make sure he got the ticket and would see me tomorrow, and apologized for the situation.

I'll report back tomorrow afternoon as far as what changes were made and what, if anything, we find.


Hal, if this doesn't get resolved, that is exactly what I may do. (Assuming I can't badger them to drop the fee myself.) I have to tread somewhat carefully, though. I don't want the service canned before I can port the phone number away. The bar has had the same number for like 30 years or something. I'd never hear the end of it if that number got lost.



This is exactly what is happening, but now that the CO tech says the circuit looks terrible.... I'm not sure



I know it isn't our IP pool, clearly the one PC test bore that out. The only two IP's on the internal side would have been the router and the PC. This is why the above statement by ffej010 resonated with me. Maybe our CO is tapped out.

Chris, sometimes a business has to suck it up and pay the price for good service. With cable so cheap now a days. just have them move over.

Unfortunately we are a food service business that has been hit really hard by the pandemic. Right now we are barely breaking even. The biggest hurdle is the ETF and the Installation fees. The actual service is fairly comparable in price.

This is their last chance to make it right, and they know it.

The tech was out today, and through his talks with the CO tech, they found the programming in the DSLAM (they called it "the build") was done very poorly. He got into the technical details a very little bit with me, but that's way above my pay grade. Suffice it to say, the CO tech completely scrapped the build and made a new one from scratch. Everyone is certain the problem is not a physical circuit or hardware problem. I mentioned the scenario (and even showed him this thread) of the provider running out of IP's and he didn't seem to think that was the problem, though it wasn't completely out of the question. He did mention that everyone working from home, schooling from home, and generally sticking around home, has put their entire network under considerable strain, and the weakness are showing. I liken it to the cartoon of Wile E. Coyote or one of them plugging holes in a wooden boat with their fingers and toes until they've run out of toes and fingers.

Bottom line is this...

If this fixes it, and it stays online without taking a dump every day and becomes reasonably reliable again, we will leave it be. It's not a great service, and we knew that going in, but it is adequate for our needs, as long as it stays at an acceptable level of reliability. If I go in tomorrow or the next day and it's down again.... As Major Hochstetter used to say.. "Heads will roll!"

Bonus points if you get that reference...

Chris. We do a lot of cable sales. Not sure what you mean by ETF and install fees. Usually install chargers are waved for new customers.
By ETF do you mean build fees ?

Early termination fee. We are in a three year contract with CCI.

Things to look for
1. When is contract up
2. Contract terms for CCI not being able to provide service
3. Track and document every time and for how long ckt is not providing service.
4 Est gmod amount of money lost by ckt being down.
5. CCI will not resist cancelation

Depending on who you might switch to, the new provider may do something to help buy out or buy down ETF.

Cable isn't the end-all be-all either. It can be full of problems just as you are currently experiencing....poor plant, exhausted resources, misconfigured services. That really applies to any provider and their plant and setup.

I want to kind of close out this thread. I appreciate everyone’s input. I think in a perfect world, switching to cable internet would be the more ideal solution. Neither company was willing to waive any fees, at least not without a fight, so switching providers is kind of a last resort situation.

However....

The problem does seem to be resolved. Time will truly tell, but the internet has been up and running without a single hiccup since the tech was here Wednesday. Apparently, when we upgraded to a higher speed, the CO tech at the time did a lazy hack to bump the speed without taking the time to make sure his parameters would be stable. The other CO tech totally scrapped his build and made a new one from scratch that seems to be keeping the connection stable. For now the problem is solved, and everything seems to be working as it should. I know DSL (or in our case, VDSL) is a dated technology, but it is sufficient for our needs.

Thank you all for your help, as always I really appreciate it.

Chris