Good afternoon all,
We have been experiencing a very interesting issue lately, and I figured I'd share in case any of you ran into the same issue.
Issue Overview:
A single modem causes ‘Full Carrier Utilization Signature’ (Standing Wave, Full Hay Stack) on 1,2,or in rare cases 3 upstream carriers.
We have multiple examples of the issue causing OV QOS call volume on the node. We see calls prior to the issue and during the issue, but once resolved the call volume stops.
Issue appears to happen mostly when either a single device or multiple devices go offline on a node, and goes back through the registration process.
Causes:
Full Bandwidth utilization (Standing Wave).
ServAssure shows 100% CER and < 10dBmV US SNR
Causes excessive saturation on the remaining available channels.
Dynamic Modulation happens on the impaired frequency, down to QPSK.
All devices on the node go into partial service 8x3 or 16x3, due to the impairment.
Identification:
SNR Alarms
Temporary Fix:
It appears that a single Cisco device is causing the issue.
We identify the offending device by T4 Timeouts (DOCS-IF-MIB::docsIfCmStatusT4Timeouts.2)
The offending device has T4's in the 5-7000 range over a 2-3 hour period.
A reset of the device immediately clears the issue on the node.
Root Cause:
Still investigating. Possible Cisco Software/Hardware issue.
Do you have activated Pre-Equalizer on that upstream channel?
If yes maybe you can try to disable it just for testing. Or disable and enable and look at the spectrum if some changes are happen after the reconfiguration.
Just a guess, normally Pre-Equalizer is running well nowadays.
Wittmann, That is a good suggestion, I will suggest that to our CMTS team, and see if we can give it a shot the next time this happens. Thanks again ! I will let you know if we find anything from this.
What model modem, what firmware version?
Right now we have found 3 specific model modems causing the issue:
DPQ3925C Firmware dpq3925-P15-18-c1100r5593-150316a
DPC3010i Firmware dpc3010-v302r125553-120614a
DPC 2203 Firmware dpc2203-P10-5-v202r1262-101203as
We have also experienced this issue. In our case Cisco EPC3010 modems were causing it.
We had about 3k EPC3010 in the specific plant at the time, and the problem occurred maybe once or twice per month.
A power cycle of the affected modem always solved the problem, and we never had the same modem behaving like this twice.
The problem occurred while the modems were on FW: e3000-v302r125533-110811c
Problem got solved after upgrading to FW: e3000-v302r125573-130606c
The problem has not occurred a single time since the FW upgrade (years ago by now), so my best guess is a FW issue like you mentioned.
Looks very similar to an issue where the modem was constantly sending BW-REQ packets but never sending data. It was first identified on D3.0 modems but some D3.1 still had the issue last year.
It was a chipset firmware issue, fixed by a new load.
Came across this about a year ago for the first time and 3-5 more times since. Fixed it by removing the specific upstream connector from the the cable interface and shutting it off. This breaks the upstream bonding group though, so once you fix it all the modems are in partial. Since then I've had the HFC plant techs track it instead of doing that again.