Hi
I am having trouble with the bandwidth and latency toward random CMs only during peak hours ( 19:00 ~ 22:00 +-1)
Even if there is a good connection, the utilization in all of the 8 DS channels and 4 US channels are not above 70% ( Max ) and there is no traffic done from client's side still random CMs have high ping, up to 4000 ms.
After monitoring the traffic of the mac domain's interfaces during the peak hours, i saw that all of them do not go beyond 650 Mbps.
Its not a fixed 650 but it stays at these levels. Also the chart is almost a straight line during the peak hours which says to me that there is more request for traffic but no resources to use, some kind of bottleneck.
The configuration of the mac domain is standard, as below . There are approx 1200 CM in this mac domain.
no shutdown
sync-interval 10
insertion-interval 20
ucd-interval 1000
dhcp-authorization
invited-ranging-attempts 16
initial-tech broadcast-ranging
initial-tech ofdma broadcast-ranging
no energy management
no battery backup
ip-provisioning-mode ipv4-only
no early-authentication-encryption
no upstream extended-frequency-range
cm-status event report
no multicast-dsid-forward
downstream channel bonding
upstream channel bonding
tftp-enforce
no tftp-enforce src-interface
no tftp-proxy
no upstream drop classifier
no send udc rules
no dhcp-giaddr-primary
no dhcpv6-link-address-primary
no dhcp-insert
mdd interval 2000
sid-cluster max-requests 0
sid-cluster max-outstanding-bytes 0
sid-cluster max-total-bytes 0
sid-cluster max-time 0
max sid-cluster per-service-flow 2
ip bundle 1
downstream 1 interface qam 1/4/0
downstream 2 interface qam 1/4/1
downstream 3 interface qam 1/4/2
downstream 4 interface qam 1/4/3
downstream 5 interface qam 1/4/4
downstream 6 interface qam 1/4/5
downstream 7 interface qam 1/4/6
downstream 8 interface qam 1/4/7
downstream 9 interface qam 1/5/0
downstream 10 interface qam 1/5/1
downstream 11 interface qam 1/5/2
downstream 12 interface qam 1/5/3
downstream 13 interface qam 1/5/4
downstream 14 interface qam 1/5/5
downstream 15 interface qam 1/5/6
downstream 16 interface qam 1/5/7
downstream 17 interface qam 1/6/0
downstream 18 interface qam 1/6/1
downstream 19 interface qam 1/6/2
downstream 20 interface qam 1/6/3
downstream 21 interface qam 1/6/4
downstream 22 interface qam 1/6/5
downstream 23 interface qam 1/6/6
downstream 24 interface qam 1/6/7
upstream 1 interface upstream 13/5.0/0
upstream 2 interface upstream 13/5.1/0
upstream 3 interface upstream 13/6.0/0
upstream 4 interface upstream 13/6.1/0
upstream 5 interface upstream 13/6.2/0
upstream 6 interface upstream 13/6.3/0
upstream 7 interface upstream 13/7.0/0
upstream 8 interface upstream 13/7.1/0
upstream 9 interface upstream 13/8.0/0
upstream 10 interface upstream 13/8.1/0
upstream 11 interface upstream 13/7.2/0
upstream 12 interface upstream 13/7.3/0
upstream 13 interface upstream 13/5.2/0
upstream 14 interface upstream 13/5.3/0
upstream 15 interface upstream 13/8.2/0
upstream 16 interface upstream 13/8.3/0
mgmd ipv4 query-interval 125
mgmd ipv4 version 3
mgmd ipv4 max-query-response-time 100
mgmd ipv4 proxy-interface gige 6/0
mgmd ipv4 robustness 2
mgmd ipv4 last-member-query-interval 10
mgmd ipv4 shutdown
mgmd ipv6 query-interval 125
mgmd ipv6 version 2
mgmd ipv6 max-query-response-time 100
mgmd ipv6 proxy-interface gige 6/0
mgmd ipv6 robustness 2
mgmd ipv6 last-member-query-interval 10
mgmd ipv6 shutdown
privacy kek life-time 604800
privacy tek life-time 43200
no cable cm-status report event-list 1,2,3,4,5,6,7,8,9,10,11,16,17,18,20,21,22,23,24,25,26,27
cable cm-status event 1 max-timer 2 max-number 8
cable cm-status event 2 max-timer 2 max-number 8
cable cm-status event 3 max-timer 2 max-number 8
cable cm-status event 4 max-timer 2 max-number 8
cable cm-status event 5 max-timer 2 max-number 8
cable cm-status event 6 max-timer 2 max-number 8
cable cm-status event 7 max-timer 2 max-number 8
cable cm-status event 8 max-timer 2 max-number 8
cable cm-status event 9 max-timer 2 max-number 8
cable cm-status event 10 max-timer 2 max-number 8
cable cm-status event 11 max-timer 2 max-number 8
cable cm-status event 12 max-timer 2 max-number 8
cable cm-status event 13 max-timer 2 max-number 8
cable cm-status event 14 max-timer 2 max-number 8
cable cm-status event 15 max-timer 2 max-number 8
cable cm-status event 16 max-timer 2 max-number 8
cable cm-status event 17 max-timer 2 max-number 8
cable cm-status event 18 max-timer 2 max-number 8
cable cm-status event 19 max-timer 2 max-number 8
cable cm-status event 20 max-timer 2 max-number 8
cable cm-status event 21 max-timer 2 max-number 8
cable cm-status event 22 max-timer 2 max-number 8
cable cm-status event 23 max-timer 2 max-number 8
cable cm-status event 24 max-timer 2 max-number 8
cable cm-status event 25 max-timer 2 max-number 8
cable cm-status event 26 max-timer 2 max-number 8
cable cm-status event 27 max-timer 2 max-number 8
no cm trap
cm trap interval 600
Is this an issue someone dealt with also ?
Do i need to check something else ?
Thank you for your time
Endrit
Hi,
I suggest setting the channel utilization average interval to a low period so you can see if you're really maxing out on bandwidth or not by checking it from the CMTS.
(global config)
channel-utilization-interval 5
With a low interval such as 5 seconds you can get a close-to-realtime figure to look at. After, use the "show docsis down chan util" or "show docsis up chan utilization" command to see what's reported.
On an unrelated note, I suggest configuring 'cable cm-status report event-list 1,2,3,4,5,6,7,8,9,10' on your docsis macs or the CMTS will ignore status messages from modems when partial service situations can be recovered (fec lock restored etc). Fixing that should lower the count of partial-service bonded modems over time. Ones already in partial service won't come back until rebooted.
I agree that a flatlined graph indicates you've run out of resources somewhere. It could be that not all of your channels are being bonded on (e.g. if ports are shut or the frequencies are out of the range of the modems), resulting in less available traffic. Are all 24 channels you posted in your docsis mac also configured in your service group?
Hi yoyo,
First of all thank you for your reply.
My channel-utilization-interval was 30. The utilization is equally in all of the DS ports and US ports .
The pasted below its not of the peak hours.
0/0/0 (114000000 Hz) 55.6 39 283 18
0/0/1 (122000000 Hz) 55.6 39 9 275
0/0/2 (130000000 Hz) 55.6 39 3 282
0/0/3 (138000000 Hz) 55.6 39 4 281
0/0/4 (146000000 Hz) 55.6 40 20 260
0/0/5 (154000000 Hz) 55.6 40 12 270
0/0/6 (162000000 Hz) 55.6 41 16 271
0/0/7 (170000000 Hz) 55.6 41 17 271
0/1/0 (178000000 Hz) 55.6 39 240 53
0/1/1 (186000000 Hz) 55.6 39 24 264
0/1/2 (194000000 Hz) 55.6 41 29 268
0/1/3 (202000000 Hz) 55.6 42 17 275
0/1/4 (210000000 Hz) 55.6 46 19 260
0/1/5 (218000000 Hz) 55.6 37 17 264
0/1/6 (226000000 Hz) 55.6 43 16 263
0/1/7 (234000000 Hz) 55.6 38 11 269
0/2/0 (242000000 Hz) 55.6 37 270 49
0/2/1 (250000000 Hz) 55.6 38 11 283
0/2/2 (258000000 Hz) 55.6 37 12 278
0/2/3 (266000000 Hz) 55.6 37 11 278
0/2/4 (274000000 Hz) 55.6 44 21 276
0/2/5 (282000000 Hz) 55.6 38 19 273
0/2/6 (290000000 Hz) 55.6 48 21 266
0/2/7 (298000000 Hz) 55.6 44 32 249
12/1.0 (48600000 Hz) 15.4 15 27 134
12/1.1 (36600000 Hz) 15.4 10 42 125
12/1.2 (41000000 Hz) 15.4 9 67 96
12/1.3 (44200000 Hz) 15.4 12 50 114
I changed it to 5 just so i can see in a real time the utilization as you said.
My load-balance configuration is based on the utilization.
I also configured cable cm-status report event-list 1,2,3,4,5,6,7,8,9,10.
All of the 24 ports are enabled. Frequency Range ( 114 MHz - 298 Mhz ).
I use Docsis 2.0 TWG850-4 and Docsis 3.0 CBN CH6643E, both accept 108 - 862 (TWG) - 1002 ( CBN ) MHz based on their datasheets.
The 24 ports are configured within the service group.
service group 12.1
qam 0/0/0
qam 0/0/1
qam 0/0/2
qam 0/0/3
qam 0/0/4
qam 0/0/5
qam 0/0/6
qam 0/0/7
qam 0/1/0
qam 0/1/1
qam 0/1/2
qam 0/1/3
qam 0/1/4
qam 0/1/5
qam 0/1/6
qam 0/1/7
qam 0/2/0
qam 0/2/1
qam 0/2/2
qam 0/2/3
qam 0/2/4
qam 0/2/5
qam 0/2/6
qam 0/2/7
upstream 12/1.0
upstream 12/1.1
upstream 12/1.2
upstream 12/1.3
My Uplink on SMMs are 10Gbps links.
Please let me know if you need anything else
Endrit.H
I apologize as this was unclear - was the utilization you posted during peak hours, or not during peak hours? Most downstream channels were around 40% utilization which is fine. If see upwards of 80-90% during peak hours that will be a problem. Also, 30 second port averages are okay. I suggested the 5 second averages just for checking for spot/instant bursts of data. You'll want to leave it at 30.
Hey yoyo,
That was not during the peak hours. I switched that back to default, to 30 s. When i left it for a 5 second , i observed that some DSs would go up to 89-91 %.
I may think that this could be a problem for the modems who are connected to the DS who reach 90 % but this doesn't solve the problem for those modems who are connected to other DSs and still having high latency and trouble generating traffic.
0/0/0 (114000000 Hz) 55.6 60 377 11
0/0/1 (122000000 Hz) 55.6 62 2 386
0/0/2 (130000000 Hz) 55.6 61 5 383
0/0/3 (138000000 Hz) 55.6 64 7 386
0/0/4 (146000000 Hz) 55.6 76 15 266
0/0/5 (154000000 Hz) 55.6 72 8 273
0/0/6 (162000000 Hz) 55.6 79 10 272
0/0/7 (170000000 Hz) 55.6 68 11 274
0/1/0 (178000000 Hz) 55.6 89 220 60
0/1/1 (186000000 Hz) 55.6 87 19 266
0/1/2 (194000000 Hz) 55.6 84 17 263
0/1/3 (202000000 Hz) 55.6 91 16 266
0/1/4 (210000000 Hz) 55.6 81 31 248
0/1/5 (218000000 Hz) 55.6 59 22 276
0/1/6 (226000000 Hz) 55.6 67 14 282
0/1/7 (234000000 Hz) 55.6 78 19 261
0/2/0 (242000000 Hz) 55.6 55 225 65
0/2/1 (250000000 Hz) 55.6 57 18 281
0/2/2 (258000000 Hz) 55.6 50 34 321
0/2/3 (266000000 Hz) 55.6 62 17 264
0/2/4 (274000000 Hz) 55.6 53 22 259
0/2/5 (282000000 Hz) 55.6 60 23 266
0/2/6 (290000000 Hz) 55.6 51 31 250
0/2/7 (298000000 Hz) 55.6 50 34 246
12/0.0 (51000000 Hz) 15.4 12 17 17
12/0.1 (47000000 Hz) 15.4 11 21 10
12/1.0 (51000000 Hz) 15.4 36 28 147
12/1.1 (42200000 Hz) 15.4 31 39 127
12/1.2 (37800000 Hz) 15.4 34 86 81
12/1.3 (47000000 Hz) 15.4 29 39 133
12/2.0 (50400000 Hz) 15.4 16 62 71
12/2.1 (47000000 Hz) 15.4 10 41 99
12/2.2 (43800000 Hz) 15.4 23 35 100
12/2.3 (33600000 Hz) 15.4 15 39 101
12/3.0 (51000000 Hz) 15.4 18 43 69
12/3.1 (47000000 Hz) 15.4 19 34 84
12/3.2 (39000000 Hz) 15.4 19 40 75
12/3.3 (43000000 Hz) 15.4 24 48 71
12/4.0 (51000000 Hz) 15.4 7 30 14
12/4.1 (47000000 Hz) 15.4 7 23 25
12/5.0 (51000000 Hz) 15.4 18 69 26
12/5.1 (47000000 Hz) 15.4 17 38 61
12/6.0 (41200000 Hz) 15.4 46 78 119
12/6.1 (44400000 Hz) 15.4 51 85 107
12/6.2 (38000000 Hz) 15.4 44 64 137
12/6.3 (47600000 Hz) 15.4 49 54 142
...
13/12.0 (51000000 Hz) 15.4 8 10 21
13/12.1 (47000000 Hz) 15.4 7 21 6
13/13.0 (41600000 Hz) 15.4 29 76 59
13/13.1 (33800000 Hz) 15.4 25 77 49
Thank you for your time.
Endrit
Hi, which software release are you running on the C10G?
show version
show system
Then other commands would be helpfully to get a overview about your system:
Maybe to much cable modems on the linecards:
show cable modem downstream module summary
show cable modem upstream module summary
More information about the affected MAC Domain, in this example MAC Domain 1, Service-Groups A and B, modulation-profiles 3,4,5
show interface docsis-mac 1
show interface docsis-mac 1 topology
show bonding-group mac-domain 1
show downstream channel set mac-domain 1
show upstream channel set mac-domain 1
show service group A
show service group A verbose
show service group B
show service group B verbose
show modulation-profile 3
show modulation-profile 4
show modulation-profile 5
Hi wittmann,
Attached you will find the information.
The docsis mac configuration is same to all as i posted above. The same goes to service groups as well.
Only the downstream/upstream channels are different.
I had bonding groups configured but it seemed that they worsen the problem somehow so i removed them.
The modulation-profile configuration is left default.
Thank you for your time.
Endrit
Alright, the good news is that you're running one of the latest software releases ;)
Also the distribution of the cable modems seems be moved fair across the line cards.
I have to admit that I have no experience on the C10G platform which such a kind amount of upstream channels on one DOCSIS MAC Domain. You should keep in mind that one upstream channel will consume a bitrate on the MAP Messages of 250kbps each if this upstream is idle. This bitrate is mirrored across all primary capable downstream channels of the MAC Domain. So in your case with 26 upstream channels you have a idle bitrate of approx. 6,5 Mbps on each downstream channel. If the upstream is in use from the cable modems this bitrate will go higher because the CMTS will fill up the MAPs with the Grants for the upstream bursts. Not sure how much this will rise, it was never necessary to take a closer look on the setups which I know with a maximum of eight upstreams per MAC Domain.
Maybe your issue is based on some inefficient upstream settings.
During the Topology View I would say you have at least a return path spectrum with up to 65 MHz. In this case, with this informations I would say you should be able to go with 6.4 MHz upstreams instead of 3.2 MHz. Or is there some other service available on the upstream spectrum?
With a wider upstream channel you could change the minislot size on the upstream configuration perhaps from 4 to 2 which means the upstream is a little bit more efficient.
Further some informations make no sense for me:
Example:
US Cable Chan Oper Chan Channel Mini Mod Power Service
Int Mac ID State Type Freq(Hz) Width Slot Prof (.1dBmV) Group(s)
12/1.0/0 1 3 UP atdma 51000000 3200000 4 3 100 12.1
12/1.1/0 1 4 UP atdma 47000000 3200000 4 3 100 12.1
12/1.2/0 1 23 UP atdma 47000000 3200000 4 3 100 12.1
12/1.3/0 1 24 UP atdma 39000000 3200000 4 3 100 12.1
So there is the upstream 12/1.1 and 12/1.2 which is the same RF-Port but same frequency at different upstream channels?
I guess this will work because of using different service-groups but if the CMTS will accept this config, I would say the burst receiver will have only errors!
Also the C10G is not a DOCSIS 3.1 CMTS, of course the Software is but not the Hardware. So I'm wondering why I see a DOCSIS 3.1 OFDM downstream in the upstream channel set:
1 273 12/3.2/0-12/3.3/0, 0/0.0w
I guess this is just a view bug on the CMTS.
Btw. check that no DOCSIS 3.1 is enabled by the software.
You should not get a output with the command:
show run | inc "cable docsis version 31"
If you got an output deactivate it:
config
no cable docsis version 31
Hi,
Can you paste:
show cable modem sum total
Upstream map size
Show upstream signal-quality - during peak hours
Hello Tony
CASA-C10G#show cable modem downstream module summary
Module Total Active Registered Secondary Offline
Modems Modems Modems Modems Modems
qam 0 3339 2954 2951 14644 385
qam 1 3174 2836 2835 13713 338
Total: 6513 5790 5786 28357 723
CASA-C10G#show cable modem upstream module summary
Module Total Active Registered Secondary Offline
Modems Modems Modems Modems Modems
ups 12 2830 2503 2501 4766 327
ups 13 3683 3289 3288 5945 394
Total: 6513 5792 5789 10711 721
upstream map size 2
SNR is between 32-40 in all of the US ports.
Thank you for your time
Endrit
Hello wittmann,
I changed the upstream map size from 2 to 5.
Having 26 - 32 US in a mac domain apparently produces up to 16000 MAPs/s in each DS so I am thinking its not really necessary to have an upstream map size 2.
Can you tell me in what manual you saw that one upstream channel will consume a bitrate on the MAP Messages of 250kbps each if this upstream is idle ?
Is there a way to calculate how much of the BW is reserved due to MAPs or any kind of communication between cable modem and CMTS ?
Regarding the spectrum , we have some limitations so we cant use frequencies above 55000000 Hz.
Also in order to keep a more stable SNR we chose 64qam on US with a 3.2MHz channel-width.
When switching some of the USs to 6.4 MHz we face problems time to time with the SNR.
We are looking forward to improve the coaxial network , but as you may have encountered it is a slow process.
I am thinking of splitting the DMs into 16 DS , and using less US per DM as an urgent solution.
I tried splitting the DM into 2 other DMS where the first 12 DS were primary and 12 others were secondary and vice versa on the 2nd DM , but i faced another problem with my modems not registering or talking too long to register.
Regarding the upstream , i reconfigured it. It was in working state already because the CMTS change the frequency adapting to the optimum SNR levels.
We do not have any ofdm enable, it must be a view bug.
Also the docsis version 31 is not enabled
Thank you for your time,
Endrit
No manual, just own research with a DOCSIS RF-Protocol analyser (DP-360) based on MAP size of 2 ms
Btw. when did you change the MAP size from to 2 ms to 5 ms? Before or after you're facing those latency issues?
Do you not think that a reduce of the MAP size, which also reduces the grants for the upstream opportunities will increase such problems like latency and jitter? Especially with this amount of cable modems in your DOCSIS MAC Domains.
My recommendation at all: Even if it's possible that the Casa System can support such a lot upstream channels per MAC Domain, I won't get the risk to use such a setup. I would guess there is a reason that Cisco CMTS limits the upstream channels per MAC Domain to 8 and on Arris CMTS I think the limit is 12.
Just my two cents ;)
Hello wittman
I changed upstream map size after facing the problem.
The problem is that having 26-30 US per mac domain would generate too much MAPs so i increased the value in order hoping to have more bandwidth for data traffic.
Ping was not affected, at least not that much to be noticed and to be concerned.
I also changed the dynamic map advance from 200 to 600 in the upstream settings.
I did 2 tests.
1.
I split MAC Domain 1 ( 1100 modems ) into MAC Domain 10 and MAC Domain 11. I had 24 DS x 26 US.
I then used 24 DS x 12 US for MAC domain 10.
On MAC Domain 11 i used 24 DS of another Line Card i had and 14 US of what remained.
I connected 2 optical transmitters,one for each mac domain. In this case i have in total 48 DS x 26 US.
What i saw was that the traffic surpassed 900 Mbps.I had no problem in the peak hours with the ping or traffic requests.
2.
I split MAC Domain 5 (1000 modems ) into MAC Domain 12 and MAC Domain 13. I had 24 DS x 12 US.
I then used 16 DS x 8 US ( 740 modems ) for MAC Domain 12.
I used 8 DS x 4 US ( 260 modems ) for MAC Domain 13.
I used only one optical transmitter.
I saw that the traffic for both of MAC Domains would not go above 650 Mbps ( so its same as it was before splitting).
My question :
Is it mandatory that every MAC Domain should have its own optical transmitter ? Or can it be a transmitter limitation , in your experience ?
On both tests was used the same type of transmitter.
Thank you for your time
Endrit.H
Hi,
Changing (decreasing) utilization timers is not recommended. You should only decrease it for short testing period.
Please do a small test. Select couple of modems during peak hours. When latency increases significantly, change upstream map size from 2 to 5. Check if that helps.
T.
Hello Tony
I did the test but the latency didn't change. Can you please check what i wrote to wittman and if i can have your opinion on that ?
Thank you for your time!
Endrit.H
Hello Endrit,
I will say 1 transmitter 1 Mac Domain (2 or more transmitters in 1 Mac Domain yes, 1 transmitter in 2 Mac Domain no)
and after you create bundles of upstream depending of your receivers.
For your investigation, I propose that you setup easydocsis (cloud version) in order to help you.
I will be pleased to assist you this way.
Good luck
oss-iris
Well...
I would need to get my hands on CLI to check what is going on...
Do you have valid support agreement with CASA? They have access to pretty cool tools :)
Hi to all
What I did was dividing the mac domains and adding new optical transmitters.
So the mac domains with 24 DS and 26 US(or more), were split into 2 or 3 other mac domains with 16 DS per each Mac domain and not more than 12 US.
We used one optical transmitter per each new mac domains that were created.
We observed that the traffic in Total was increased by almost 1 Gbps.
I guess the main problem was having more than 12 US per mac domain will decrease a lot the downstream bandwidth due to MAPs resevation.
Also changing the upstream map size from 2 to 5 was an option i changed. It didnt affect the ping or jitter to that level to be problematic.
Thank you all for your time, your comments and valuable thoughts on my problem
Endrit.H
One more thing... Are you using BPI?
Hi Tony
Yes i am using BPI.
Endrit.H
Kalo ne BSR64K