| Attachment | Size |
|---|---|
 10k-crashlog.txt | 6.69 KB |
We've starting having issues with one of out 10k CMTSs. Over the past week SNMP has been spotty but the CPU utilization isn't above 50%. We detected a few drops in traffic and after troubleshooting we found that 1 of the 10k line cards would crash. I looked through the logs and see a lot of errors with slot 5/0 talking to the PRE. I'll attach the tail of the logs before the card crashes.
We've reloaded(hw-module subslot 5/0 shutdown) the line card and reinserted it thinking it was a physical connection issue but that hasn't worked. We have a card on it's way to swap out with but I wanted to check with some folks here to see if you've had any issues like this before? We don't have support so I can't open a TAC case.
I appreciate any advice.
Thanks!
What IOS version are you running ?
We have SCG and its seems nice and stable
A few trains back we were having problems with 20x20 crashes. The CPU on one would go high and then the others would usually follow.
Some troubleshooting suggestions :
You can see linecard stats with commands like
* show controllers cable 5/0/0 proc-cpu
* show controllers cable 5/0/0 mem-stats
you can also log into the cards :
* telnet to the cmts, then :
* telnet 127.0.0.50 (for linecard 5/0)
* you then can poke around, show log (etc)
We're running SCF3. I have SCG6 that I could try to upgrade too. I like that telnet command, very useful! I will look around though logs, memory and CPU to see if something stands out.
Thank you!
Checked back through my notes. We had the 20x20 crashing on SCF3 and SCF4. (MALLOC and IPC errors, high LC CPU)
Believe the bug was fixed in SCF5, but we jumped to SCG2 which came out about the same time. (Jan 2013)
Haven't seen the bug since then
I'm upgrading one 10k to SCG6. Interesting you talk about high LC CPU. I recently started graphing all CPUs including all line cards and can see where the LC will run consistently at ~80% CPU load. I didn't think it was normal so maybe that's what is causing it.
EDIT: I saw all those errors that you talked about, not just high LC CPU.
Thanks for the input!