I have two (out of two) c100G chassis that keep randomly rebooting starting last week. Been running for years previously without issue. Nothing shows up in the log as to why, just running along as normal and then hard resets with 'WA-CLI-1: smm6: cfg_recover_smm():351: pid = 652, tid = 12057, err = Success. start recover SMM configurations ...' being the next log entry.
They both have redundant SMMs and each chassis reboots every time within a minute or two of each other.
They are in separate buildings, towns, and power grids so it is not facility related, but they are in the same public IP block. They are running older firmware (7.2.6) and we have no way of obtaining newer, so I am wondering if anyone knows about any vulnerabilities or something that might point me in the right direction.
Apparently, this has been a problem for a week now on many CMTS CASA is experiencing random restarts.
I'm having the problem on a "CASA-systems device C10G" - Release 7.2.5, Ver 3, build63f2 - 10GIGE: x2 - GIGE: x8
I've heard, but I don't know how true it is, that there's a problem with 7x firmware. The problem is definitely external; I'm testing various solutions, we'll see.
I have 2 customers with 7.x software with the same symptoms (both SMM 2x10G and SMM 8x10G), but 8.x seems to be unaffected. Highly recommend moving to an 8.x train of software. Note: you need SMM8x10G's to move to 8.x, SMM 2x10G's have been EOLed and have no support in 8.x trains, meaning 7.2.6 is the highest version you can go on that hardware, and they still seem affected.
Potential cause of the problem -- Talking to a friend of mine reminded me of a potential reason for the reboots everyone is seeing.
netconf port is open allowing attackers to overwhelm the SMM with authentication requests. blocking port TCP port 830 inbound to the CMTS should mitigate the vulnerability.
In the older versions of Code I don;t think you can turn off nor block the traffic at CMTS.
you can see if this affects you by running tcpdump in diag and filter for port 830. See if you are getting lots of ssh to port 830.If so, this is likely the culprit.
I have the SMM 8x10G's but no way of obtaining 8.x firmware. any ideas?
Glad I am not crazy.
According to me, the problem with 7.x firmware is ICMP packets. Why? I've always had this ACL on every CASA CMTS.
IP access-list server_host
permit ICMP type 8 any any
permit all "$management network" any any any
permit tftp any any
deny all any any any any
I just cut off traffic to the CMTS on the edge router. It's been fine since yesterday, but I'm still checking to see if my joy is premature. We'll see.
PS. Can someone share the 8.x firmware? I have 8.2.6.0 and 8.6.1.1, but I haven't tested it because I don't have anything to test it on.
from release notes
Distributed Denial of Service operational changes
The ddos command for enabling mitigation of Distributed Denial of Service (DDoS)
breaches has been enhanced with the drop-threshold parameter to support
configuration of the DDoS trigger threshold. The packet drop threshold was
previously fixed at 500 Mbps per service flow and can now be set in the range 1 to
300000 Mbps depending on customer requirements for declaring sessions under
attack. The default setting is 500 Mbps.
We’re seeing the same problem on two C40G units with SMM _2x10G. After years of stable operation, they now crash within seconds of each other.
Has anyone identified the root cause—or found a fix?
It sounds like an overflow problem to a certain port. See posts above. May have to block it at your edge
While checking our other customers to see if any of them may be affected, we ran across one that is running Rel 7.2.6, Ver 1,build8c0a, Tue Dec 1 16:37:55 EST 2020 and they do not seem to be running into the problem, their chassis has been up for 128 weeks.
They do have a proper access-class applied to their chassis with a deny all any any any at the end.
ip access-list REMOTE
! all their permits for remote management
remark "Drop everything else"
deny all any any any any
access-class in REMOTE
I am not sure if the exploiters have not found them, the access-class is working or the more recent version of code is what is preventing them from having issues. But most of the users experiencing the issue I have seen are using a 7.x code from 2018/2019
Ideally, you want to upgrade to 8.x have not seen any 8.x cmts with the issue. But beware 8.x code requires SMM 8x10G or greater, DS 8x192 cards and US 16x8 cards at a minimum the SMM 2x10G, DS 8x96 and US 16x4 cards are not supported.
From 8.8.3 release notes:
Supported hardware with Release 8.8
The following hardware is supported with this release:
• C100G or C40G chassis.
• SMM 8x10G, SMM300G or SMM300Gm modules.
• UPS 16x8 module — Supports one OFDMA channel and four ATDMA channels
per port; eight ATDMA channels without OFDMA.
• CSC Rev 11 and Rev 12 cards for Casa DAA (Remote PHY) services.
• CSC 2x8 I/O cards for Casa DAA (Remote PHY) services.
• QAM 8x192 module — Two OFDM and 64 SC-QAM channels per port.
• Video and shared channels:
— Combined 64 multi-port shared channels (broadcast) and 64 narrowcast
channels per port: 1
— Up to 80 narrowcast channels for DOCSIS, SDV, VOD. (All 128 multi-port
shared channels must be narrowcast with no shared broadcast.)
— With up to 64 narrowcast channels (0 to 63) for DOCSIS
— With up to 80 narrowcast channels (0 to 79) for SDV, VOD
— Maximum multi-port shared (broadcast) channels: 64
— Up to 24 Annex A, B, and C channels per port operating in DVB Simulcrypt
encryption mode. These 24 channels must be configured at the upper channel
range, such as 40 to 63. All shared channels support DVB Simulcrypt.
• BDM, BDM_204MHZ, BDM2m_204MHZ, and BDM I/O module.s
• 10G EPON (DPoE) modules.
Note: Release 8.8 software operates on the C100G, C40G, and the DA1000,
DA1250. DA2000, and DA2200 Remote PHY nodes. Note that Release 8.8 is
NOT supported on any other Casa hardware platforms.
Ok shameless plug time, if you need to upgrade your DS/US or SMM cards to upgrade we have DS8x192 and SMM300GM in stock to help you out, sales <at> rocnetsupply.com
In summary, the attack on the CMTs CASA is coming from the internet, probably through ICMP. This is the ACL on the CMTs CASA I've always had, and it allowed these restarts. But when I blocked everything to the CMTS on the edge router, it helped. So, I probably need to remove ICMP from it.
IP access-list server_host
permit icmp type 8 any any
permit all MGMT_CLASS any any any
permit tftp any any
deny all any any any any
!
!
access-class in server_host