There is a weird issue going on on one of our CMTS's that has me stumped. There are several customers complaining that the internet is down or taking a long time to load. What I have been able to duplicate is just on some sites, the pages cannot load. This is a Cisco ubr7246VXR, MC-28U, NPE-G1. We do run BGP right on the CMTS and connect to AT&T. We are only receiving default and partial routes from them.
From my laptop hooked to a cable modem on a six foot plant in the headend, I was unable to load a few sites. Yahoo brings up just the header in the browser tab but never loads. Godaddy brings up the header and sits there and tries to load, if I hit the stop button, I have the majority of the page. Google comes up just fine as well as many other sites. Others will not load. Speedtest won’t load. I’ve tried using different DNS servers, same issues. I moved my laptop to the GigE interface and can browse just fine. I originally thought DNS issue, but the sites are resolving just fine and are pinging fine and traceroute is OK from my laptop.
There is a pattern going on of multiple pings from the CMTS to the different sites. I ran several pings from the CMTS to multiple sites and after 15 pings, it doesn’t work. Also, a traceroute from the CMTS to Yahoo’s first IP got to their network, but at hop 13 or 14 all I get is asterisks. I don't think this is an RF issue, but I could be wrong. All signal levels are good, SNR is around 38, modem downstream receive is -1 to 1 db on most modems and upstream power is between 30 and 40 db on most modems. I do have a couple modems that have 8 - 14 db downstream receive and upstream power of 15 to 21 db. A tech has been dispatch to pad those, but those wouldn't cause issues with other modems, would the?
I am stumped and have no idea where to look. I have another CMTS running off the same AT&T router that we peer with, and there are no issues there, so I don’t think it’s an AT&T issue. Does anyone have any ideas?
Sounds like your upstream is running too hot. your situation may be special of course, but seems to me you can afford to pad down your upstreams like 6dB...in addition to that the modems running at 15 to 21 dB upstream are coming back way too hot to the server and can easily be causing upstream errors across the node, too many will cause exactly the issue you are having.
If you gave your upstream SNR levels, would help confirm this as well.
I would be eliminating the cable network from any test. Do all your initial testing from the CMTS.
So you have 3 gige ports on the CMTS and one is used for the uplink to AT&T? You plugged your laptop in the 2nd one, chucked a /30 or similar on it and could surf the web fine?
You telnetted into the CMTS and tried ping/traceroute to the internet and were having problems after 15 or so pings? This sounds a bit strange! Perhaps try experimenting with the ping source-ip, try the gige IP vs the cable bundle IPs
What sort of connection is it to AT&T? A metro ethernet? Full gige speed, or rate-limited? Can you ping the far end of the link without troubles? Is the link maxing with traffic?
Can you ping AT&T at full 1500 MTU ?
Is the BGP session stable? (not flapping)
Are you doing any NAT?
We ended up fixing the issue. It was a BGP issue. The CPE block was not being announced to AT&T, just via EIGRP over a lower speed microwave link to our core network where the aggregate block is being announced. Pings worked just fine. The connection to AT&T is fiber. Traceroutes didn't point to the issue, it was when we did a reverse traceroute into the bundle interface that we saw the problem. Everything is working great now. Thanks for all the help.