Home
Port-channels, vPCs, and Spanning Tree Issues

Carole Warner ReeceIn the last week I talked to two different customers about issues with their port-channels, vPCs, and Spanning-Tree Protocol. The first customer thought he had a STP issue between his N7Ks and his N5Ks, but he actually had vPC design issue. The second customer also had the same design issue, but did not know about it until I pointed it out to him. Since I thought it was interesting that both had the same issue, I thought I would write up a summary.


Broken Design
The broken design looks like the following diagram:

vPC and Port-Channel Issue

The same set of VLANs are supported on both port-channels. Do you see the issue?

Both customers wanted to have a lot of bandwidth between the N7K and the N5K pairs. However, their network was not forwarding traffing on all the links. The show spanning-tree vlan 10 command should help illustrate the problem:

N7K1# sh span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 Cost 1
 Port 4109 (port-channel14)
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address 0026.980a.8d42
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1         128.4096 (vPC peer-link) Network P2p
Po11             Root FWD 1         128.4109 (vPC) P2p
Po12             Altn BLK 1         128.4110 (vPC) P2p
. . .
N7K1



N7K2# sh span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 Cost 2
 Port 4096 (port-channel1)
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address f866.f210.8ea2
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 1         128.4096 (vPC peer-link) Network P2p
Po11             Root FWD 1         128.4109 (vPC) P2p
Po12             Altn BLK 1         128.4110 (vPC) P2p
. . .
N7K2#



N5K1# sh span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 This bridge is the root
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address 0005.73ba.8abc
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10             Desg FWD 1         128.4096 (vPC peer-link) Network P2p
Po11             Desg FWD 1         128.4109 P2p
. . .
N5K1#



N5K2# show span vlan 10

VLAN0010
 Spanning tree enabled protocol rstp
 Root ID Priority 32778
 Address 0005.73ba.8abc
 Cost 1
 Port 4096 (port-channel1)
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

 Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
 Address 0005.73bc.de3a
 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10             Root FWD 1         128.4096 (vPC peer-link) Network P2p
Po12             Desg FWD 1         128.4110 P2p
. . .
N5K2#

 

Logical View
So the design issue is that the two port-channels between the N7K and the N5K layers are creating a Layer 2 loop, and STP is appropriately blocking one set of interfaces from forwarding traffic.

Logical View of vPCs, Port-Channels, and STP Issue

Please note that replacing the port-channels on the N5Ks with vPCs on N5K will NOT resolve the issue -- if you have the same VLANs on both vPCs you will still have blocking on the VLANs.

 

Recommended Design
What both customers need to do is put all the links between the two Nexus pairs into one large vPC. With this small design changes, all links will be forwarding between the devices:

Recommended Design vPC, Port-Channels, and STP

 

vPC and Port-Channel Operations
I offer some observations from my migration tests of vPCs and port-channels on Nexus devices in case you ever need to migrate port-channels to vPCs on a production network:

  • When a single interface is configured as a port-channel, the interface resets. (I saw a loss of 4 packets on a long ping test across the link, and the log file shows the interface down for 5 seconds.)
  • When an interface is added to an operational port-channel, the interface resets but the port-channel and existing interfaces are not reset.
  • When a port-channel is reconfigured as a vPC, the port-channel and its interface resets. This is disruptive, even when trying to minimize the impact I saw a loss of 7 packets in my tests.  (I had configured the secondary switch in the vPC pair with a shutdown interface on the member port, put it in the vPC, then configured the primary switch. The primary switch went down. I hopped back to secondary switch, no shut the interface, and saw that the secondary switch was up in 7 seconds. In my test, the primary switch came up 15 seconds after it initially went down.)
  • When the vPCs are not configured on both peers, the VLANs on the port-channel are suspended. When I removed the port-channel from the secondary switch, then reapplied it, the physical port on the primary vPC switch never went down, but when the VLANs were suspended devices lost connectivity.
  • When a peer member link is added to an existing vPC (such as enabling right link when left link is up), the left link stays up.


Minimizing Downtime
My customers have not yet had a chance to schedule a maintenance window to correct their designs. Depending on which port-channel is blocked, they may take a slightly diferent order in the steps to migrate to the back-to-back vPC.  For either case, they will need to create the new vPC on both N5Ks, and move the ports into the new vPC on the N5K2. They will also have to move the ports out of vPC 12 into  vPC11 on the N7Ks.

-- cwr

_____________________________________________________________________________________________

If you would like some additional details on vPCs, the following references may be helpful: 

Comments (10)Add Comment
feedSubscribe to this comment's feed
0
Interesting
written by Carsten, July 24, 2012
Very interesting, thank you for your insight. How would this look on the configuration side? Do you have the 7ks in the vpc domain 1 and the 5ks in the vpc domain 100, both pairs using vpc11 on their port-channels? Or are all 4 devices in the same vpc domain?
Carole Warner Reece
reply to Carston
written by Carole Warner Reece, July 24, 2012

The vPC domain ID should be different for each pair.

The N7Ks are in vPC domain 1, the N5Ks remain in vPC domain 100. Cisco doc say that the reason for this distinction is that the system ID is derived from the MAC address of the switch, and for vPC this MAC address is derived from the domain ID. So in a back-to-back vPC configuration, if the neighboring switches use the same domain ID, a system ID conflict may arise in the LACP negotiation, leading to an unsuccessful LACP negotiation.

Carole
0
...
written by Carsten, July 24, 2012
So we actually are talking about 2 port-channels with vPC: one port-channel consisting of the 4 interfaces of the N7ks and one port-channel consisting of the 4 interfaces of the N5ks. That is because both N5ks and both N7ks act as one device.
Makes sense.
Thanks again.
Carole Warner Reece
reply to Carsten
written by Carole Warner Reece, July 24, 2012
Correct - one virtual port channel with 4 interfaces is configured on the N7Ks, and a second virtual port channel with 4 interfaces is configured on the N7Ks. The vPCs use the same port-channel number, and vPC number on both the N7Ks and the N5Ks.
0
...
written by Eric, July 25, 2012
Would you have to add the vpc 11 statement on the port-channels on both the 5k and 7k's in a vPC?
Carole Warner Reece
Reply to Eric
written by Carole Warner Reece, August 13, 2012
Yes, the "vpc 11" statement is needed for both vPC peer devices. So the two N5Ks (as well as the two N7Ks) would need the "vpc 11" command to put all four interfaces in the same port-channel.
0
...
written by George, September 14, 2012
But what is the point or benefit of having a vPC link between the 5Ks? Would the design not be valid (without any STP blocked ports) without that link?
Carole Warner Reece
Reply to George
written by Carole Warner Reece, September 17, 2012
George -
Both customers had a Layer 2 peer-link supporting their server VLANs between the N5Ks so they could support east-to-west traffic between their servers across the N5Ks.
Yes, if there was no Layer 2 Link between the N5Ks then there would be no Layer 2 blocking on Po11 or Po12. In this case, traffic between servers on different N5Ks would have to go up to the N7Ks to get to the other N5K.
Carole
0
...
written by Craig, November 16, 2012
Hi Carole

Great info, I was testing this a while back and saw the same results as your customers but couldn’t get it working properly and ended up creating a loop in the network every time I tried to get this working

I’m thinking that the basic config would be:

7K1
interface port-channel 11
vpc 11
7K2
interface port-channel 11
vpc 11

5K1
interface port-channel 11
vpc 11
5K2
interface port-channel 11
vpc 11

I’m also thinking that the port channel numbering isn’t important so long as the vpc numbers are the same across the switches in each domain?


Thanks Craig
Carole Warner Reece
Reply to Craig
written by Carole Warner Reece, February 12, 2013
Yes, the VPC numbers must match in each domain, but do not need to match across the two domains.
You might want to look over one of the references, http://www.netcraftsmen.net/re...tches.html , which shows configuration commands.

Write comment

busy