I can't for the life of me get a L3 LACP link to come up on the PRE5, but only on the Active PRE.
Active PRE is A (t0/0/x)
If I have a LACP with t0/0/0 and t0/1/0 (first port on both PRE) only t0/1/0 gets added to the port-channel
if I have a LACP with t0/0/0 and t0/0/1 (first 2 ports on active PRE) no ports get added (both in susp state), port-channel interface stays down
If I have a LACP with t0/1/0 and t0/1/1 (first 2 ports on standby PRE) both ports are added
Other end is a Juniper MX204, Config is identical on all ports on both ends.
In all cases the individual ports come up/up (eg in the second case, t0/0/0 and t0/0/1 will both come up/up, but the Port-channel will be down/down)
Cisco Side: (all ports are configured the same, currently wired ports are 0/0/0 and 0/0/1 on the Active PRE5)
interface Port-channel2
ip address xxx.xxx.xxx.xxx 255.255.255.254
no negotiation auto
interface TenGigabitEthernet0/0/0
no ip address
channel-group 2 mode passive
#sh lacp 2 int
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
Channel group 2
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te0/0/0 SP susp 32768 0x2 0x0 0x1 0x7C
Te0/0/1 SP susp 32768 0x2 0x0 0x2 0x7C
Juniper Side:
interfaces {
xe-0/0/3:0 {
gigether-options {
802.3ad ae2;
}
}
xe-0/0/3:1 {
gigether-options {
802.3ad ae2;
}
}
(snip)
ae2 {
description "20G LAG to xxxx";
aggregated-ether-options {
link-speed 10g;
lacp {
active;
periodic slow;
}
}
unit 0 {
family inet {
address xxxxxx/31;
}
}
}
show lacp interfaces ae2 extensive
Aggregated interface: ae2
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/3:0 Actor No Yes No No No Yes Fast Active
xe-0/0/3:0 Partner No Yes No No No Yes Fast Passive
xe-0/0/3:1 Actor No Yes No No No Yes Fast Active
xe-0/0/3:1 Partner No Yes No No No Yes Fast Passive
LACP protocol: Receive State Transmit State Mux State
xe-0/0/3:0 Defaulted Fast periodic Detached
xe-0/0/3:1 Defaulted Fast periodic Detached
LACP info: Role System System Port Port Port
priority identifier priority number key
xe-0/0/3:0 Actor 127 08:05:e2:fe:b8:48 127 3 3
xe-0/0/3:0 Partner 1 00:00:00:00:00:00 1 3 3
xe-0/0/3:1 Actor 127 08:05:e2:fe:b8:48 127 4 3
xe-0/0/3:1 Partner 1 00:00:00:00:00:00 1 4 3
show interfaces terse | match ae2
xe-0/0/3:0.0 up up aenet --> ae2.0
xe-0/0/3:1.0 up up aenet --> ae2.0
ae2 up down
ae2.0 up down inet xxxx/31
I have tried both slow and fast PDU, and changing which side is active, as well as trying active/active with no changes, only the ports on the standby PRE5 get added to the Port-Channel interface
any ideas?
Some ideas :
Do you have > 10G of WAN licenses on the PRE5?
"show license"
Do you have the B2B cables installed/working?
"show int ten0/0/0 b2b status"
PRE5 redundancy working properly?
"show redundancy"
-------
Our PRE5 are uplinking to Juniper EX switch at 60G or 80G, no drama
CMTS :
interface Port-channel1
description ** switchname1:ae4 (80G)
mtu 1600
no ip address
load-interval 30
no negotiation auto
interface TenGigabitEthernet0/0/0
description ** switchname1:xe-0/0/6
mtu 1600
no ip address
load-interval 30
channel-group 1 mode active
etc
JUNIPER :
switchname1> show configuration interfaces ae4
description "cmtsname1:port-c1 (80G)";
traps;
mtu 1600;
aggregated-ether-options {
link-speed 10g;
lacp {
active;
}
}
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members [ VLAN-X VLAN-Y VLAN-Z ];
}
}
}
switchname1> show configuration interfaces xe-0/0/6
description "cmtsname1:ten0/0/0 (ae4 member)";
traps;
ether-options {
802.3ad ae4;
}
Licensed for 4x10G:
cmts#show lic
Index 1 Feature: WAN_License
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: 4/4/0 (Active/In-use/Violation)
License Priority: Medium
B2B links are all up:
cmts#show int t0/0/0 b2b stat
PREA B2B port 0 link status: UP
PREB B2B port 0 link status: UP
cmts#show int t0/0/1 b2b stat
PREA B2B port 1 link status: UP
PREB B2B port 1 link status: UP
cmts#show int t0/0/2 b2b stat
PREA B2B port 2 link status: UP
PREB B2B port 2 link status: UP
cmts#show int t0/0/3 b2b stat
PREA B2B port 3 link status: UP
PREB B2B port 3 link status: UP
And Redundancy is in SSO:
cmts#show red
Redundant System Information :
------------------------------
Available system uptime = 12 weeks, 1 day, 23 hours, 32 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = SSO
Operating Redundancy Mode = SSO
Maintenance Mode = Disabled
Communications = Up
If it weren't for the fact that it seems to work fine on the standby SFP+ ports, I would think it were the knockoff 40g->4x10g breakout cable, but the same legs and optics of the breakout work fine on the standby, with the same config.
I am wondering if it is something to do with the PRE5 WAN backhaul ports being able to be in 'either backup or load sharing mode' and I have no idea how to check what mode they are in.
Seems a bit strange.
I agree that cisco mentions the backup mode vs load-sharing mode.
I was never able to find any documentation on this, I just configured mine in a big port-channel (which i assume is load-sharing mode).
What happens if you temporarily put /30 onto each tengig link (rather than putting them into LAG)
Do they come up and ping OK?
What happens if you failover the active PRE?
What IOS version on the CMTS?
Well, that gets me somewhere but don't know how to fix it, de-aggregating and putting a /30 on each link gets them to come up/up, but the active PRE ports will not pass traffic (and will not ping through)
Running 12.2(33)SCJ2a
I can try a failover in a maintenance window, this CMTS has active customers on it.
EDIT:
I spun up a new port to a cisco switch using first-party Cisco optics, and the behavior is the same.
Curiously on the switch side I see both the switch and the CMTS interface in the ARP table, but on the CMTS side, I do not see the switch
Switch#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.222.222.17 - 881d.fc66.xxxx ARPA TenGigabitEthernet1/5
Internet 10.222.222.18 2 001f.ca20.xxxx ARPA TenGigabitEthernet1/5
cmts#sh arp t0/0/2
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.222.222.18 - 001f.ca20.xxxx ARPA TenGigabitEthernet0/0/2
On the Standby PRE ARP shows both ends
cmts#sh arp t0/1/0
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.222.222.10 - 001f.ca20.xxxx ARPA TenGigabitEthernet0/1/0
Internet 10.222.222.9 2 0805.e2fe.xxxx ARPA TenGigabitEthernet0/1/0
For the active PRE, it sounds like there is only one-way dataflow (cmts->switch)
So I don't think it's a LACP specific problem, that is just a side effect of this one-way dataflow.
Do the SFP light levels look OK on the CMTS eg ?
show int ten0/0/0 controller | incl power:
Tx power: -2.8 dBm (5205 in units of 0.1 uW)
Rx power: -1.9 dBm (6391 in units of 0.1 uW)
We are using 12.2(33)SCJ4
Are you using same SFP brand/model in both PRE5?
Maybe compare the outputs of
show int ten0/0/0 controller | begin CR10K5 BH SFP
If it were me, I would probably do the failover to other PRE, and then physically eject and reseat the problematic PRE5
It gets weirder...
I have not only failed over which PRE is active, and reseated the problem PRE, But have completely replaced that PRE with a spare and the issue persists.
Additionally if I put the original PRE into a different chassis, and load identical IOS and the full config from the live CMTS, both PREs act as they should.
So I am getting one way data on the SFP+ Ports of whatever is in Slot A, but that seems to be the only thing effected, I had 1000's of modems running with slot A as the active PRE with no issues.
That is making me think it could be chassis related, but I have never heard of anything like this, and a forklift replace of that chassis is going to be a nightmare.
OK, well I would reload next
If you have SCJ4, do that at same time