I'm having a fit out of what seems like only 16 channel modems getting stuck in an offline state (assuming not locking to downstream) after disruptions such as small blips in the power or the plant losing and re-establishing RF. The are Pace D5001 wireless combo routers, and SB6183 wired modems. Sometimes it is taking a few reboots on the customer end to get the modems to get back online on the system. If the modem isn't rebooted, it will sit indefinitely in the offline state on the CMTS, and in whatever hung up state it is in at the customers house. Our 8 channel mix of modems (DCP3010, SBG6580, SB6141, SB6182) don't seem to be affected.
Posting up my modem and part of the CMTS config for a known affected node, and would love any suggestions about what I might be missing before I pull all of my hair out!
Modem Config:
Network Access Control:on
Upstream Service Flow Encodings
Service Flow Reference:1
Quality of Service Parameter Set:provisioned admitted active
Upstream Maximum Sustained Traffic Rate:6880000
Downstream Service Flow Encodings
Service Flow Reference:2
Quality of Service Parameter Set:provisioned admitted active
Downstream Maximum Sustained Traffic Rate:113000000
SNMP MIB Object(docsDevNmAccessStatus.1):1.3.6.1.2.1.69.1.2.1.7.1, Integer, 4
SNMP MIB Object(docsDevNmAccessIp.1):1.3.6.1.2.1.69.1.2.1.2.1, IP Address, 255.255.255.255
SNMP MIB Object(docsDevNmAccessIpMask.1):1.3.6.1.2.1.69.1.2.1.3.1, IP Address, 0.0.0.0
SNMP MIB Object(docsDevNmAccessCommunity.1):1.3.6.1.2.1.69.1.2.1.4.1, Octet String, XXXXX
SNMP MIB Object(docsDevNmAccessControl.1):1.3.6.1.2.1.69.1.2.1.5.1, Integer, 2
SNMP MIB Object(docsDevNmAccessInterfaces.1):1.3.6.1.2.1.69.1.2.1.6.1, Octet String, @
SNMP MIB Object(docsDevNmAccessStatus.2):1.3.6.1.2.1.69.1.2.1.7.2, Integer, 4
SNMP MIB Object(docsDevNmAccessIp.2):1.3.6.1.2.1.69.1.2.1.2.2, IP Address, 255.255.255.255
SNMP MIB Object(docsDevNmAccessIpMask.2):1.3.6.1.2.1.69.1.2.1.3.2, IP Address, 0.0.0.0
SNMP MIB Object(docsDevNmAccessCommunity.2):1.3.6.1.2.1.69.1.2.1.4.2, Octet String, XXXXX
SNMP MIB Object(docsDevNmAccessControl.2):1.3.6.1.2.1.69.1.2.1.5.2, Integer, 3
SNMP MIB Object(docsDevNmAccessInterfaces.2):1.3.6.1.2.1.69.1.2.1.6.2, Octet String, @
Enable 2.0 mode:on
Downstream Channel List
Downstream Frequency Range
Downstream Frequency Range Start:99000000
Downstream Frequency Range End:201000000
Downstream Frequency Range Step Size:6000000
Baseline Privacy Configuration Settings
Authorize Wait Timeout:10
Reauthorize Wait Timeout:10
Authorization Grace Time:600
Operational Wait Timeout:10
Rekey Wait Timeout:10
TEK Grace Time:600
Authorize Reject Wait Timeout:9
SA Map Wait Timeout:1
SA Map Max Retries:4
Maximum Number of CPEs:1
CMTS Config:
cable load-balance docsis-group FN 15 MD Cable6/0/0
method utilization us-method modems
policy pure-ds-load
cable load-balance docsis-group FN 16 MD Cable6/0/0
method utilization us-method modems
policy pure-ds-load
!
cable load-balance docsis-group 114
downstream Integrated-Cable 6/0/0 rf-channel 0-3
downstream Integrated-Cable 6/0/1 rf-channel 0-3
downstream Integrated-Cable 6/0/2 rf-channel 0-3
downstream Integrated-Cable 6/0/3 rf-channel 0-3
upstream Cable6/0/0 0-3
!
cable load-balance docsis-group 115
downstream Integrated-Cable 6/0/0 rf-channel 0-3
downstream Integrated-Cable 6/0/1 rf-channel 0-3
downstream Integrated-Cable 6/0/2 rf-channel 0-3
downstream Integrated-Cable 6/0/3 rf-channel 0-3
upstream Cable6/0/0 4-7
!
controller Integrated-Cable 6/0/0
rf-channel 0 cable downstream channel-id 49
rf-channel 0 frequency 111000000 annex B modulation 256qam interleave 32
rf-channel 0 rf-power 45.0
no rf-channel 0 rf-shutdown
rf-channel 1 cable downstream channel-id 50
rf-channel 1 frequency 117000000 annex B modulation 256qam interleave 32
rf-channel 1 rf-power 45.0
no rf-channel 1 rf-shutdown
rf-channel 2 cable downstream channel-id 51
rf-channel 2 frequency 123000000 annex B modulation 256qam interleave 32
rf-channel 2 rf-power 45.0
no rf-channel 2 rf-shutdown
rf-channel 3 cable downstream channel-id 52
rf-channel 3 frequency 129000000 annex B modulation 256qam interleave 32
rf-channel 3 rf-power 45.0
no rf-channel 3 rf-shutdown
!
controller Integrated-Cable 6/0/1
rf-channel 0 cable downstream channel-id 53
rf-channel 0 frequency 135000000 annex B modulation 256qam interleave 32
rf-channel 0 rf-power 45.0
no rf-channel 0 rf-shutdown
rf-channel 1 cable downstream channel-id 54
rf-channel 1 frequency 141000000 annex B modulation 256qam interleave 32
rf-channel 1 rf-power 45.0
no rf-channel 1 rf-shutdown
rf-channel 2 cable downstream channel-id 55
rf-channel 2 frequency 147000000 annex B modulation 256qam interleave 32
rf-channel 2 rf-power 45.0
no rf-channel 2 rf-shutdown
rf-channel 3 cable downstream channel-id 56
rf-channel 3 frequency 153000000 annex B modulation 256qam interleave 32
rf-channel 3 rf-power 45.0
no rf-channel 3 rf-shutdown
!
controller Integrated-Cable 6/0/2
rf-channel 0 cable downstream channel-id 57
rf-channel 0 frequency 159000000 annex B modulation 256qam interleave 32
rf-channel 0 rf-power 45.0
no rf-channel 0 rf-shutdown
rf-channel 1 cable downstream channel-id 58
rf-channel 1 frequency 165000000 annex B modulation 256qam interleave 32
rf-channel 1 rf-power 45.0
no rf-channel 1 rf-shutdown
rf-channel 2 cable downstream channel-id 59
rf-channel 2 frequency 171000000 annex B modulation 256qam interleave 32
rf-channel 2 rf-power 45.0
no rf-channel 2 rf-shutdown
rf-channel 3 cable downstream channel-id 60
rf-channel 3 frequency 177000000 annex B modulation 256qam interleave 32
rf-channel 3 rf-power 45.0
no rf-channel 3 rf-shutdown
!
controller Integrated-Cable 6/0/3
rf-channel 0 cable downstream channel-id 61
rf-channel 0 frequency 183000000 annex B modulation 256qam interleave 32
rf-channel 0 rf-power 45.0
no rf-channel 0 rf-shutdown
rf-channel 1 cable downstream channel-id 62
rf-channel 1 frequency 189000000 annex B modulation 256qam interleave 32
rf-channel 1 rf-power 45.0
no rf-channel 1 rf-shutdown
rf-channel 2 cable downstream channel-id 63
rf-channel 2 frequency 195000000 annex B modulation 256qam interleave 32
rf-channel 2 rf-power 45.0
no rf-channel 2 rf-shutdown
rf-channel 3 cable downstream channel-id 64
rf-channel 3 frequency 201000000 annex B modulation 256qam interleave 32
rf-channel 3 rf-power 45.0
no rf-channel 3 rf-shutdown
!
interface Cable6/0/0
description |Rocky|Roddy|
downstream Integrated-Cable 6/0/0 rf-channel 0-3 upstream 0-7
downstream Integrated-Cable 6/0/1 rf-channel 0-3
downstream Integrated-Cable 6/0/2 rf-channel 0-3
downstream Integrated-Cable 6/0/3 rf-channel 0-3
cable mtc-mode
cable shared-secret 7 XXXXX
no cable packet-cache
cable bundle 1
cable upstream max-ports 8
cable upstream bonding-group 214
upstream 0
upstream 1
upstream 2
upstream 3
attributes 80000000
cable upstream bonding-group 215
upstream 0
upstream 1
attributes 80000000
cable upstream bonding-group 216
upstream 4
upstream 5
upstream 6
upstream 7
attributes 80000000
cable upstream bonding-group 217
upstream 4
upstream 5
attributes 80000000
cable upstream 0 connector 0
cable upstream 0 frequency 20000000
cable upstream 0 channel-width 3200000 3200000
cable upstream 0 ingress-noise-cancellation 400
cable upstream 0 description Rocky
cable upstream 0 docsis-mode scdma
cable upstream 0 spreading-interval 16
cable upstream 0 codes-per-minislot 4
cable upstream 0 active-codes 112
cable upstream 0 power-adjust continue 10
cable upstream 0 range-backoff 3 6
cable upstream 0 modulation-profile 323
cable upstream 0 equalization-coefficient
no cable upstream 0 shutdown
cable upstream 1 connector 0
cable upstream 1 frequency 23500000
cable upstream 1 channel-width 3200000 3200000
cable upstream 1 ingress-noise-cancellation 400
cable upstream 1 description Rocky
cable upstream 1 docsis-mode scdma
cable upstream 1 spreading-interval 16
cable upstream 1 codes-per-minislot 4
cable upstream 1 active-codes 112
cable upstream 1 power-adjust continue 10
cable upstream 1 range-backoff 3 6
cable upstream 1 modulation-profile 323
cable upstream 1 equalization-coefficient
no cable upstream 1 shutdown
cable upstream 2 connector 2
cable upstream 2 frequency 27000000
cable upstream 2 channel-width 3200000 3200000
cable upstream 2 ingress-noise-cancellation 400
cable upstream 2 description Rocky
cable upstream 2 docsis-mode scdma
cable upstream 2 spreading-interval 16
cable upstream 2 codes-per-minislot 4
cable upstream 2 active-codes 112
cable upstream 2 power-adjust continue 10
cable upstream 2 range-backoff 3 6
cable upstream 2 modulation-profile 323
cable upstream 2 equalization-coefficient
no cable upstream 2 shutdown
cable upstream 3 connector 2
cable upstream 3 frequency 30500000
cable upstream 3 channel-width 3200000 3200000
cable upstream 3 ingress-noise-cancellation 400
cable upstream 3 description Rocky
cable upstream 3 docsis-mode scdma
cable upstream 3 spreading-interval 16
cable upstream 3 codes-per-minislot 4
cable upstream 3 active-codes 112
cable upstream 3 power-adjust continue 10
cable upstream 3 range-backoff 3 6
cable upstream 3 modulation-profile 323
cable upstream 3 equalization-coefficient
no cable upstream 3 shutdown
cable upstream 4 connector 4
cable upstream 4 frequency 17000000
cable upstream 4 channel-width 3200000 3200000
cable upstream 4 ingress-noise-cancellation 400
cable upstream 4 description Roddy
cable upstream 4 docsis-mode scdma
cable upstream 4 spreading-interval 16
cable upstream 4 codes-per-minislot 4
cable upstream 4 active-codes 112
cable upstream 4 power-adjust continue 10
cable upstream 4 range-backoff 3 6
cable upstream 4 modulation-profile 324
cable upstream 4 equalization-coefficient
no cable upstream 4 shutdown
cable upstream 5 connector 4
cable upstream 5 frequency 20200000
cable upstream 5 channel-width 3200000 3200000
cable upstream 5 ingress-noise-cancellation 400
cable upstream 5 description Roddy
cable upstream 5 docsis-mode scdma
cable upstream 5 spreading-interval 16
cable upstream 5 codes-per-minislot 4
cable upstream 5 active-codes 112
cable upstream 5 power-adjust continue 10
cable upstream 5 range-backoff 3 6
cable upstream 5 modulation-profile 324
cable upstream 5 equalization-coefficient
no cable upstream 5 shutdown
cable upstream 6 connector 6
cable upstream 6 frequency 25000000
cable upstream 6 channel-width 6400000 6400000
cable upstream 6 ingress-noise-cancellation 400
cable upstream 6 description Roddy
cable upstream 6 docsis-mode scdma
cable upstream 6 spreading-interval 16
cable upstream 6 codes-per-minislot 4
cable upstream 6 active-codes 112
cable upstream 6 power-adjust continue 10
cable upstream 6 range-backoff 3 6
cable upstream 6 modulation-profile 324
cable upstream 6 equalization-coefficient
no cable upstream 6 shutdown
cable upstream 7 connector 6
cable upstream 7 frequency 31400000
cable upstream 7 channel-width 6400000 6400000
cable upstream 7 ingress-noise-cancellation 400
cable upstream 7 description Roddy
cable upstream 7 docsis-mode scdma
cable upstream 7 spreading-interval 16
cable upstream 7 codes-per-minislot 4
cable upstream 7 active-codes 112
cable upstream 7 power-adjust continue 10
cable upstream 7 range-backoff 3 6
cable upstream 7 modulation-profile 324
cable upstream 7 equalization-coefficient
no cable upstream 7 shutdown
cable privacy mandatory
no keepalive
hold-queue 4096 in
!
interface Cable6/0/1
no cable packet-cache
cable upstream max-ports 0
!
interface Cable6/0/2
no cable packet-cache
cable upstream max-ports 0
!
interface Cable6/0/3
no cable packet-cache
cable upstream max-ports 0
!
interface Cable6/0/4
no cable packet-cache
cable upstream max-ports 0
!
interface Integrated-Cable6/0/0:0
description Service Group 28 - Channel 16
cable bundle 1
cable rf-bandwidth-percent 10
!
interface Integrated-Cable6/0/0:1
description Service Group 28 - Channel 17
cable bundle 1
cable rf-bandwidth-percent 10
!
interface Integrated-Cable6/0/0:2
description Service Group 28 - Channel 18
cable bundle 1
cable rf-bandwidth-percent 10
!
interface Integrated-Cable6/0/0:3
description Service Group 28 - Channel 19
cable bundle 1
cable rf-bandwidth-percent 10
!
interface Integrated-Cable6/0/1:0
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/1:1
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/1:2
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/1:3
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/2:0
description Service Group 12 - Channel 96
cable bundle 1
cable rf-bandwidth-percent 50
!
interface Integrated-Cable6/0/2:1
description Service Group 12 - Channel 97
cable bundle 1
cable rf-bandwidth-percent 50
!
interface Integrated-Cable6/0/2:2
description Service Group 12 - Channel 98
cable bundle 1
cable rf-bandwidth-percent 50
!
interface Integrated-Cable6/0/2:3
description Service Group 12 - Channel 99
cable bundle 1
cable rf-bandwidth-percent 50
!
interface Integrated-Cable6/0/3:0
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/3:1
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/3:2
cable bundle 1
no cable rf-bandwidth-percent
!
interface Integrated-Cable6/0/3:3
cable bundle 1
no cable rf-bandwidth-percent
!
interface Wideband-Cable6/0/0:0
description Service Group 27 - Wideband 16x
cable bundle 1
cable rf-channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel 3 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 3 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 3 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 3 bandwidth-percent 1 remaining ratio 100
!
interface Wideband-Cable6/0/0:1
description Service Group 27 - Wideband 8x
cable bundle 1
cable rf-channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel 3 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 3 bandwidth-percent 1 remaining ratio 100
!
interface Wideband-Cable6/0/0:2
description Service Group 27 - Wideband 8x
cable bundle 1
cable rf-channel controller 2 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 3 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 3 channel 3 bandwidth-percent 1 remaining ratio 100
!
interface Wideband-Cable6/0/0:3
description Service Group 27 - Wideband 4x
cable bundle 1
cable rf-channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel 3 bandwidth-percent 1 remaining ratio 100
!
interface Wideband-Cable6/0/0:4
description Service Group 27 - Wideband 4xtra-1
cable bundle 1
cable rf-channel controller 1 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 1 channel 3 bandwidth-percent 1 remaining ratio 100
!
interface Wideband-Cable6/0/0:5
description Service Group 27 - Wideband 4xtra-2
cable bundle 1
cable rf-channel controller 2 channel 0 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 1 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 2 bandwidth-percent 1 remaining ratio 100
cable rf-channel controller 2 channel 3 bandwidth-percent 1 remaining ratio 100
!
cable fiber-node 15
description Rocky
downstream Integrated-Cable 6/0/0 rf-channel 0-3
downstream Integrated-Cable 6/0/1 rf-channel 0-3
downstream Integrated-Cable 6/0/2 rf-channel 0-3
downstream Integrated-Cable 6/0/3 rf-channel 0-3
upstream Cable 6/0 connector 0 2
!
cable fiber-node 16
description Roddy
downstream Integrated-Cable 6/0/0 rf-channel 0-3
downstream Integrated-Cable 6/0/1 rf-channel 0-3
downstream Integrated-Cable 6/0/2 rf-channel 0-3
downstream Integrated-Cable 6/0/3 rf-channel 0-3
upstream Cable 6/0 connector 4 6
!
We saw some 16x4 Broadcom modems have a bug where they get stuck offline with DS LED flashing. Modem had to be power cycled to get back online.
After a lot of investigation, we found 2 different causes :
PROBLEM 1.
========
One of our 16x4 modems had a bug reloaded to load-balance.
When the CMTS used DCC init-tech 1, the modem would get stuck offline.
Our initial workaround was to alter the load-balance stanza to have "init-tech-list 2-4" (ie exclude init-tech 1)
Later, the vendor applied a Broadcom patch to prevent the issue (they explained the patch related to "multi-threads mutex locks")
Reproduce steps (probably need to do in lab) :
Disable D3 US channel bonding (I usually just temp remove the upstream connector from fiber-node staza)
Then try these commands to manually initiate DCC and see if you get a hang
get the current US and DS
scm 48f7.c073.ea96 primary
in this example the US is 3, and the DS is In9/0/4:16
MAC Address IP Address Host MAC Prim Num Primary DS
Interface State Sid CPE Downstream RfId
48f7.c073.ea96 172.20.129.48 C9/0/4/U3 online(pt) 355 2 In9/0/4:16 29712
keep the same primary, but move to US2, using DCC method 1
test cable dcc 48f7.c073.ea96 integrated-Cable 9/0/4:16 2 1
our result was hang <=== THIS IS THE BUG
keep the same primary, but move to US2, using DCC method 2
test cable dcc 48f7.c073.ea96 integrated-Cable 9/0/4:16 2 2
our result was OK
keep the same primary, but move to US1, using DCC method 3
test cable dcc 48f7.c073.ea96 integrated-Cable 9/0/4:16 1 3
our result was OK
keep the same primary, but move to US0, using DCC method 4
test cable dcc 48f7.c073.ea96 integrated-Cable 9/0/4:16 0 4
our result was OK
keep the same primary, but move to US1, using DCC method 0
test cable dcc 48f7.c073.ea96 integrated-Cable 9/0/4:16 1 0
our result was OK
PROBLEM 2.
========
On our cBR8 CMTS (not on uBR10k), there is a bug where Packet Sequence Number (PSN) has a random number rather than 0 after a new DSID is created (eg at initial registration or after load balance). This seems to cause the modem to sometimes (not always) get stuck.
We recently reported the bug to Cisco, and it has been assigned bug ID CSCvt28555, due to be patched in 17.3.1w and 16.12.1z
But since you are using uB10k you probably can ignore this issue.
ALSO
====
Also I have some suggestions regarding your CMTS config, I will try and find some time and post info on that
So I was able to swing by a customer actively having this issue today, and have a hunch on part of what might be going on. This sub was on Node 16, which uses US 4-7, but the modem had DS lock with upstream hang. Looking in the modem stats page it was stuck trying to broadcast against US SID 3 (Interface cable upstream 2) to come online, which is not a valid US port for that node.
Somewhere in the config have a feeling I am going to find a lack of isolation on the affected nodes between US groups. I'm still open to suggestions on CMTS optimizations mbowe (as always, you've got some good suggestions) as this may only wind up being some of the issue as well.
I'm also going to grab one of the affected modem types and try "PROBLEM 1" as these are most definitely broadcom chipset units, and want to get ahead of that.
Thanks again for the input!
Yes i have seen that lack of isolation issue before too !
What is the typical US PWR being reported by modems in these nodes? If it is low eg 20s and 30s then you should add more headend attenuation to increase isolation. ideally the modems will have USWPR in the low 40s.
You did say it sometimes takes several power cycles to get a modem back online, so that matches this same theory. Whereas my 2 x bugs are always fixed with 1 power cycle.
Also there would be different LED combinations for your isolation issue vs my bugs
Mine are stuck with DS flashing
Yours would be locked DS with US flashing
Found part of our issues, and it indeed was port isolation being too low through the telemetry network between fiber nodes. Worked on resolving that issue by moving transmitter attenuation to after the fiber node splitters and have part of our issues resolved. Now higher spec modems (16x4 and up) are managing to come back online near 100% of the time when they lose power and all RF.
Our typical US power range is 38 - 47, depending on season and pre or post balancing for the season.
Only issue I have remaining is these modems are managing to hang in an event where downstream service stays online, but upstream service is interrupted long enough for a modem to go back to trying to come online. Restoring reverse path after the modem has started trying to range the upstream results in the modems staying offline until either downstream or power is interrupted.
Slow progress, lol.
Thanks for the update
So as an ongoing update to this catastrophe, we are finding that the 16 and 24 channel Broadcom based modems being used in our setup are failing to continue initial ranging if the upstream path is missing more than after 90 seconds after the start of upstream ranging. No solution exists to recover them after 90 seconds except power cycling the modems of breaking forward signal to the modems.
I can watch the ranging bursts on a spectrum analyzer on these modems and at 90 seconds it just stops trying to send additional ranging bursts to the CMTS. No other modems (docsis 2, 4 channel broadcom, 8 channel broadcom, and 24 channel puma 6) exhibit this issue, and will automatically heal once upstream is restored no matter the amount of time they are in this state.
CMTS is running Version 12.2(33)SCJ4, modem chipsets affected are BCM33843 and BCM3384.
I'm drawing at straws here, but has anyone else ran across this one?
That shouldnt happen
We have UBR10K with same IOS as you, and 16 and 24 channel Broadcom modems. No problem.
What is the modem model and firmware version?
What is the modem's base Broadcom firmware eg 5.7.1mp4
Have you tried telnet or serial into affected modem to see what they are doing ?
My 24 channel BCM3348 gateway is a Pace D5001, looks like it is running a Broadcom 5.7.1 payload, Pace Gateway software load on this one is 2.168.2.40:
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.1.0 = STRING: "Oct 23 2015, 14:15:13(-0400)"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.3.0 = STRING: "eCos"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.4.0 = STRING: "Net-SNMP 5.0.9"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.1 = STRING: "BFC System"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.2 = STRING: "BFC System"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.3 = STRING: "Standard EmbeddednTarget Support for BFC"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.4 = STRING: "eCos BFC Application Layer"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.5 = STRING: "CM DOCSIS Application"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.6 = STRING: "Broadcom Data-Only CM Vendor Extension"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.7 = STRING: "Broadcom Linux Residential/Voice Gateway"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.8 = STRING: "Broadcom Linux Residential/Voice Gateway Apps Partition"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.9 = STRING: "eRouter Application"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.10 = STRING: "Broadcom eRouternCustomer Extension"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.1 = STRING: "5.7.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.2 = STRING: "5.7.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.3 = STRING: "3.0.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.4 = STRING: "3.0.2"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.5 = STRING: "5.7.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.6 = STRING: "3.0.2"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.7 = STRING: "2.6.30-1.7.1-pace-2.168.2.40"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.8 = STRING: "1.7.1-pace-2.168.2.40"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.9 = STRING: "5.7.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.10 = STRING: "3.0.2"
My 16 channel BCM33483 modem is an Arris SB6183, looks like it is running a Broadcom 5.5.10mp5 payload, Arris software load on this one is D30CM-OSPREY-1.5.2.5-GA-00-NOSH:
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.1.0 = STRING: "May 20 2016, 08:01:19 (-0700)"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.3.0 = STRING: "eCos"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.4.0 = STRING: "Net-SNMP 5.0.9"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.1 = STRING: "BFC System"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.2 = STRING: "BFC System"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.3 = STRING: "Standard Embedded Target Support for BFC"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.4 = STRING: "eCos BFC Application Layer"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.5 = STRING: "CM DOCSIS Application"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.6 = STRING: "Motorola Data-Only CM Vendor Extension"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.1 = STRING: "5.5.10mp5"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.2 = STRING: "5.5.10mp5"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.3 = STRING: "3.0.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.4 = STRING: "3.0.2"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.5 = STRING: "5.5.10mp5"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.6 = STRING: "3.0.0a"
I think the old Broadcom software is your problem. You need to get up to latest 5.7.1mp4
PACE D5001
No experience with this model sorry
SB6183
The next newer firmware after
D30CM-OSPREY-1.5.2.5-GA-00-NOSH (Broadcom 5.5.10mp5)
is
D30CM-OSPREY-2.4.0.0-GA-03 (Broadcom 5.7.1mp4)
I'll see if I can get with my contact at Commscope and get a firmware update to load for the SB6183. Ever since the merger of Arris into Commscope our SSO portal has been royally screwed up, and I guess procrastinating to get that fixed has bit me in the butt, lol.
::EDIT:: Loading D30CM-OSPREY-2.4.0.0-GA-03 onto an SB6183 now and running the test procedure against it!
Moved the SB6183 over to a newer firmware (D30CM-OSPREY-2.4.0.0-GA-03-NOSH), and still experience the same issues as before the update:
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.1.0 = STRING: "Nov 14 2018, 22:21:55(-0800)"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.3.0 = STRING: "eCos"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.4.0 = STRING: "Net-SNMP 5.0.9"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.1 = STRING: "BFC System"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.2 = STRING: "BFC System"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.3 = STRING: "Standard EmbeddedTarget Support for BFC"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.4 = STRING: "eCos BFC Application Layer"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.5 = STRING: "CM DOCSIS Application"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.2.6 = STRING: "Motorola Data-Only CM Vendor Extension"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.1 = STRING: "5.7.1mp4"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.2 = STRING: "5.7.1mp4"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.3 = STRING: "3.0.1"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.4 = STRING: "3.0.2"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.5 = STRING: "5.7.1mp4"
SNMPv2-SMI::enterprises.4413.2.2.2.1.9.1.1.5.1.3.6 = STRING: "3.0.0a"
mbowe, you wouldn't happen to know off hand the hex integer for "SNMPv2-SMI::enterprises.4413.2.2.2.1.1.1.1.0" to open up telnet to LAN as well would you? I've tried a few of the values I would use on other models with no luck. I'm going to try logging into telnet and watching the modem after it hangs.
We finally found a resolution to this issue. Turns out this broadcom chipset cannot handle running 32QAM and SCDMA at the same time. We went through 3 different IOS revisions and two entirely unique hardware plants, with the only thing in common being the config they shared, both of which exhibited the issue.
Coming off SCDMA onto ANY modulation depth in ATDMA only mode across all 4 channels resolved the issue. Wound up having a more burst resistant ATDMA profile designed, and subbing it in from SCDMA operation.