| Using FEX with the F2 Card in a Nexus 7000 |
|
A previous blog covered the F2 card -- see http://www.netcraftsmen.net/resources/blogs/looking-at-the-cisco-nexus-7000-f2-card.html. It covered the technology, and the key item was that I've heard that the F2 will only do FCoE and perhaps a couple of other features with the soon-to-be-released Sup2 and new NXOS code. For FEX support, check the documentation and check with Cisco, not all FEX were "supported initially". This is an area where things are happening quickly, and the preliminary slideware may have gaps or not be 100% complete and current. The F2 + FEX RuleNexus 7000's require you to connect a FEX to a N7K with one or more port-channels. That's probably a good practice anyway -- if you connect the FEX with one link, you may want to add links to it with minimum disruption later. If you connect up with one port-channel, you can't get in trouble -- well, at least not with the F2 + FEX Rule. The rule is that if you connect the FEX to the N7K F2 card with more than one port-channel, the connecting port channels must use the same ports relative to the port-groups. Let's go through that a piece at a time. The SoC ASIC chips on the F2 have port groups consisting of four consecutive port, i.e. 1-4, 5-8, etc. (For those who got used to 1,3,5,7 and 2,4,6,8 on the M1 cards, well, you'll have to learn a different pattern.) If you have 1 port group of 4 ports to say a 2248, then using F2 ports 1-4 or 5-8 works. If you have 2 port groups of 2, then using two port-channels to F2 ports 5 and 6, and 9 and 10 works. That's because 5 and 6 are the first two ports for their port group, and so are 9 and 10. Here's a diagram showing this: Here's an example using the first port of 4 port groups:
And here's one showing a connection pattern that is not supported:
If you add another FEX, it does NOT have to match that pattern -- but you have to follow the rule for the port-channels going to it. Suppose you have a FEX with two port-channels on F2 ports 5+6 and also 9+10. You can hook another up with a port-channel on say ports 8, 12, 16, and 20 (port 4 in their respective port groups).
If you're connecting up a Nexus 2232, you could do something like one port channel to ports 21, 22, 25, 26, 29, 30, 33, 34. That's ports 1 and 2 in each port group. You could not do a port channel with ports 21, 22, 23, 25, 26, 27, and 29, 30, because you've then got 3 ports, 3 ports, and 2 ports on different port groups -- that's not the same pattern on each of the port-groups used. You can split the connections across multiple F2 cards in the same chassis. The same rules apply. You can share a port-group across multiple FEX. Each FEX needs to follow the rules -- but the rules are per-FEX. That is, different FEX can use different matching patterns per port-group. So F2 ports 1 and 2 could go to one FEX, 3 to a second FEX, and port 4 to yet another FEX, if you want. See also the above diagram. The good news is it appears you'll get a mildly cryptic message if you don't follow the rules: N7K(config-if)# interface port-channel 123 N7K(config-if)# switchport mode fex-fabric So no harm done, just have to shuffle ports and where you put the port-group commands a bit. I do find myself wondering how many extra calls TAC will be getting due to this. I've heard that Microsoft tracks who coded software features and which generate the most or least support calls. And maybe ties that to programmer compensation. It would probably have to be a longer-term feedback loop. (And while not a fan of Microsoft for some of the user interface goofs, like search in Excel defaulting to "Formula" which is well-nigh useless, imagine what things might be like if they didn't have such a feedback mechanism?) I wonder if Cisco ... . Comment This!What do you think of this rule? (Be polite, please.) If you find that FCoE works without a Sup2 and the 6.1 code, please let me know or add a comment. Similarly, if you can provide actual lab feedback on what happens if you violate the FEX + F2 rule, please let me know or add a comment. It might help a fellow networking person out. Comments (12)
  written by Pete Welcher, April 30, 2012 Thanks for the comment, Michael. It's been a while since we last met / talked! East and West Coast orbits to blame for it. Anyway, I think we're in agreement, although you spotted an implication I had yet to think of. The classic example of Port Groups Gone Wild is the 6500 line cards: is it 1-12, 13-24, etc. or odds 1-23 and evens 2-24, etc. Sometimes it varies between 24 copper and 24 optical ports, if I recall correctly. And you have to know that to configure QoS without a lot of backtalk from the CLI slowing things down. (I don't want to have to know anything about the hardware when applying QoS -- or even much about the ports. Either the hardware does "enough QoS" or it doesn't. If Cisco would just give me some IF / THEN logic in the parser, or maybe a CASE statement, I could then fairly easily deal with the hardware variations so that I could actually configure a switch without having to do "show module" and then engage brain. As in IF there is a CDP neighbor, then apply INFRASTRUCTURE-LINK-QOS, etc.). Actually, one of our guys (Jim Marinelli) has achieved pretty good speedup and high accuracy by automating QoS and STP safeguard configuration across 100's of switches. So it is do-able. His logic would certainly be simpler without all the line card variations. Good point about GUI/software datacenter config tools ("SDN") and the impact of all this one-off stuff. I've certainly been close to a couple of net management products that had their hands full dealing with quirks. Heck, I just had an issue with a product that couldn't pull the per-VLAN MAC table and somehow therefore couldn't figure out what VLANs were on a certain switch model and IOS code rev level. Of course, the poorly-defined switch MIB multiplexing by community string (a known aberration from any remote intent of the SNMP standards) contributes to that. As do devices that have community-string@context-name as how you retrieve per-context information. Yecch! I'll stifle the rest of that rant to spare those who have read this far. Re your $1 question, I thought I said it, you can go across line cards but the same rule applies. Now what I didn't talk about at all was WHY you might want more than one port-channel per FEX. That's due to pinning control, and that's hefty enough a subject I think I'll make it another blog.
 written by Michael Butash, April 30, 2012
I must have totally missed that about the cross-module - my bad. Still sounds like bad mojo, either odd software constraints or afterthought hardware implications. Guess there's always F2.5 modules that will fix the constraint (at the cost of more internal vdc's and loopback connections - ha!).
I've not seen automation, either custom or commercial that has been able to deal with cross-platform qos effectively, so you might be money ahead there. Roll in something that can report on queue drops in hardware switchports and maybe CA will buy you (eww). I'd love to see the logic flows on how he deals with bandwidth, queuelen, tail-drops assignments and service-policy generation dynamically for different customer and model implementations! Cheers Pete, great article.
 written by Bill Greenwood, April 30, 2012
I have been unable to find any information on this restriction on CCO. Is it possible to post a link from Cisco on this restriction?
written by Pete Welcher, May 01, 2012 Cross-vendor QoS = the ultimate nightmare? Really, I think Cisco gets it in terms of functionality, mostly, for newer bigger switches (3750 models excluded). The smaller ones are quirky and can be annoyingly subject to minor differences in models (do I really want to do something different on some of the 2960 models I have in small closets?). I don't know much about the other vendor's switches, and wonder how far their QoS goes. Juniper, ok, they may get it, HP however? There are just so many takes on QoS, and from some of what I've read, there are some pretty strange misconceptions out there. Then again, I think mixing vendors leads to a lowest common denominator network, where more of your energy is going into the cross-vendor thing than to improving service levels. Doing MST for interoperability, just say no, unless you're really good at config consistency. Etc.
written by Pete Welcher, May 01, 2012 For the FEX on F2 topic, I've been recently emailed two Cisco presentations with much nicer graphics than mine. The title of one is "FEX with F2 LC Technical overview" dated back to last October. Googling that title turns up nothing. The other is a PowerPoint presentation dated 3/15/12 titled Nexus Technology Day. Neither appears to be publicly accessible yet, and Google isn't coming up with anything in my attempts to find public material about this. For the prior FCoE with F2 blog, no, I couldn't find a public reference about it, got that from Cisco SE who asked the BU for specifics.
written by Michael Butash, May 01, 2012
Cisco publicly admit hardware/feature limitations in a flagship platform? I can't imagine why it doesn't pop to the top of a google search.
 Sadly they bury these things from customers, engineers, and the public at large usually. At least until it's sold and you're trying to make it work, scratching your head, and calling TAC finally when they give you the "unpublished" ddts or internal ppt on undocumented features and limitations. I've found this far too often with cisco in the past and still. Blogs like Pete's are great places to find obscure factoids like this, just need more of them.
written by Pete Welcher, May 01, 2012
I'll grant Cisco a bit of a break on this one, it's early days yet re the F2 (for most folks). Stuff like this may get polished internally before it appears. The Nexus line has had some occasional nice "Operations Guides" pop up (although the FabricPath one vanished shortly after I spotted it -- turned up as part of L2 documentation later as nearly as I can tell).
Now re "FCoE on F2 requires Sup2", if that is indeed correct, then that's something the datasheet darn well ought to say, so people don't buy the wrong hardware. On the other hand, it's Cisco that'll have to deal with RMA on stuff that doesn't work, and ticked off customers ... I just figured I'd try to save some a possible headache caused by that one, until the datasheet gets fixed. Anyway, it seemed useful for folks to know about this little gotcha with FEX and F2. I'm tempted to repeat the mandatory product pun "it got FEXed up". But I'd never do that in public :-) And thanks for your support Mike!
 written by James, August 04, 2012 Error: Port(s) do not have the same index as existing port(s) in the fabric port-channel. Port indicies across all the asics should match. I ran into a similar issue, a 2232 connected to ports 17 and 18 over two line cards. The ports were configured with switchport mode fex-fabric, when adding the port channel to the ports on the first card it worked fine, when adding the ports on the 2nd card - same port index etc, I received the above error. Removing the switchport mode fex-fabric from the ports on the second line card, allowed me to add the port channel, which then inherited the command from the port channel.
written by Pete Welcher, August 05, 2012
See my comment to http://www.netcraftsmen.net/re...-card.html
 written by Chuah Eng Wee, May 15, 2013
I encountered the same problem with James a few days back when connecting FEX to multiple F2 card.
The logic Pete mentioned is correct. But you need to be a bit more careful when applying the configuration. Scenario: need to connect a FEX with 4 uplinks to two F2 card. you configure the first and second port with no problem. But when you configure the 3rd port, you will see the error. Why? because when you add the 3rd port, you will violate the rule : the connecting port channels must use the same ports relative to the port-groups. What you need to do is to be able to apply the configurations concurrently using, e.g, interface E3/1, E3/2, E4/1, E4/2 switchport switchport mode fex-fabric fex associate 101 channel-group 101 with this, it will work.
written by Peter Welcher, May 18, 2013 Good to know. That's yet another gotcha. I like the Nexus platforms. Overall I do have to say, there are just way too many gotchas, e.g. variations in which card supports what, CLI that only works if you magically hit the special way the programmer was thinking, etc. I understand some of that comes from the rapid development of faster and better chip sets. That doesn't mean you or have to like it. I see too much end user (or consultant) time wasted trying to get things to work. (Like having to tell a VDC it's an F2 VDC -- it can't figure that out from the interfaces I first try to allocate to it? And then bark and tell me not to mix in other line cards? Not a super big deal but the first time I did it, I wasted 15-30 minutes googling because I hadn't encountered that before. Multiply by 10 or 20 features and 10's of thousands of engineers, and lack of a little Cisco programming time has a big impact on the user community.) If people can't figure it out, they call TAC -- a back-end cost to Cisco from not doing more on the front-end coding side. I think we all agree, the engineers developing this stuff should go out of their way to avoid making the user's life harder (and avoid creating TAC calls). Based on the above reply, the engineer that developed this part of the code ran out of time or whatever but basically FAILED to do their job well (grade B-: it works, it's ugly). The feature this whole blog about I'd describe as a major user interface flaw if not a bug, since it makes the user have to know way too much about the hardware. Maybe the design team couldn't find a better way given time or cost constraints. Flaws like this are often a signed of a development team that is stretched too thin. Stuff like this needs to "just work", and that's been the awesome thing about Cisco over the years. Things like the "interface ..." syntax mean that you can pick up your skills and with a little theory, do e.g. fiber channel port channels naturally. So I try to remember that and put up with the quirks and small blemishes in the Nexus platform.
Write comment
|
Are you planning to use a Nexus 2000 FEX with one of the new F2 cards in a Nexus 7000? If so, there are some rules about how it connect it up, apparently due to limitations of the SoC (Switch on Chip) technology Cisco is using. I'll call this a "design awkwardness" rather than "bug" or "gotcha". This goes on my list of "Nexus things you need to know and remember or they might bite you some day". This one isn't a big deal, more of an "oops, got to rethink this". 
Subscribe to this comment's feed
It's also things like this that'll drive SDN developers insane trying to write automagical "make fex go here" code, and will ultimately drive the same developers away. I know guys in the cisco automation arena and it is more their job figuring out hardware quirks than anything, writing around them now. I'd like to see a .net developer try to write a sdn class for this and understand why he can't just provision a fex channel-group on port 3 and 48 let alone QoS on a 6k/7k.
So the $1 question is, will this work across modules then? If their asic implementation is that specific, it would beg to reason it would adverse affect cross-module (ala distributed) etherchannel mechanism, yet again to make these scalable/redundant outside one module.
All of these things make the hardware often feel hackish unfortunately, like someone cheated, and I think with the rise of SDN it'll only become more apparent when people try to apply logic to it. Dev's really don't want to care about the hardware architecture. These kinds of "rules" will be interesting to watch in competition like Arista's FPGA programmable switches that aren't in theory so hardware-based as firmware-based.