| Looking at the Cisco Nexus 7000 F2 Card |
|
So if you bought one planning to use it with Sup1 for FCoE, that may not work for some time (if ever?). The datasheet and At-A-Glance guide do not mention this (just that software support for FCoE is not yet available). This seems to be a gotcha waiting to bite very early adapters who haven't been talking to their Cisco SE (or whose SE isn't bugging the product team for such details). This is just a heads-up, something that might fall through the cracks. Having gotten all that out of the way, let's look at the positive side. Update (5/28/12): A follow-on blog at http://www.netcraftsmen.net/myblog-admin/using-fex-with-the-f2-card-in-a-nexus-7000.html provides additional info about hooking up a N2K FEX to the F2 card. The F2 is a Powerful Linecard!Other than that one minor detail, the F2 linecard seems fairly exciting. It provides:
Note the large MAC capability, needed for flat datacenters. More about those routing numbers: The N55xx support for 8000 IPv4 routes seems pretty good for its intended role (and enough for a small-medium core if you are OK with not having dual Sup capability). However, 32 K routes is a bit more large enterprise scale. Where I've seen potential routing table size problems for this is larger shops that didn't isolate the perimeter / partner networks. If you've got a well-separated perimeter and solid design, and are doing OSPF totally stubby or NSSA or good summarization, then 32 K routes in datacenter switches is fine. If you're less well-summarized, then moving part of your datacenter into another area for stubbiness and/or summarization might work -- or using M1 series linecards for much larger routing tables. See however below. A while back in a consulting situation I was asked how a spanning-tree loop in a Nexus VDC would affect the other VDCs. If the CPU is toast in your L2 datacenter aggregation VDC, is the core routing VDC also toasted? Fairly recently I heard from a Cisco person that the dual-core CPU (of which only one core is currently used) is time-slicing, so each VDC gets its share of the CPU. That provides some protection. Tuning the CPU time slices to match your needs that might be a nice feature to have. What's Missing?Let's look briefly at what the F2 does not do.The F2 does not do:
That's in line with Cisco's announced division of functionality, with the M-series linecards for Nexus doing the more intense L3 sorts of tasks.There are no big surprises there. One Other Little ThingIf you want your F2 card in a chassis with an M1 or F1 card, you need to put them in different VDCs. That is limiting in that you need to interconnect the VDCs with a physical cable. I'm still mulling over how I feel about that. If you were an early purchaser, you might only have M1 cards. I'd want to use the F2 for L2 functionality (and some L3), and maybe keep enough M1's for any OTV, MPLS, or LISP needs. If the M1 VDC can act as core router with the F2 VDC acting as datacenter core or aggregation, that seems to work. Designs for M1 / F1 / F2 MixThe following diagram illustrates this:
The F2 routing could interconnect VLANs, the L2 functionality would do great switching, and the L3 core router probably only has a small number of 10 Gbps or lower speed connections to other things, so having to use 10 G ports to connect the M1 VDC to the F2 VDC might not be much of a hardship. Alternative: wait for the M2 card? I've already gotten used to the idea of a separate VDC for OTV (due to the restriction that SVIs cannot be in the same VDC as OTV encapsulation). So the following seems fairly natural:
What about a mix of M1 and F1 cards? Suppose, for instance, you bought a N7K with just a 1 Gbps M1 card or two for budgetary reasons, I'd much rather have an F2 card than an F1, but then tying the M1 card to it is rather awkward. That's probably a rare situation. One positive aspect is you might be able to get a low-cost F1 card used from someone who would like to replace it with an F2 card. The other situation I can think of where you might have a mix is if you have a bunch of F1's with an M1 or two for routing off them. The challenge there is that if you put the M1 and F1's into their own VDC, you might end up with the inter-VDC cable as a bottleneck.
I'd be interested if any readers see other solutions to the latter situation, or see problems with the earlier sketches. Or have other info nuggets to contribute. Please use the Comments capability to do so (and thanks in advance). [Editorial note: Thanks to my readers for putting up with my sketches. I'm trying to liven the blog up with whiteboard-quality graphics. I'm trying out various iPad sketching tools for something that mixes ease of sketching (quick!) with good text and JPEG or PNG export (PDF? Bleh for graphics to insert into HTML). The above was done in AutoDesk SketchBookX. I may end up reverting to my PC-based IOgear digital scribe, an IR-sensed pen. I have many years of experience driving a pen. Or even Visio. Better results faster?] Comments (16)
  written by Francisco, April 12, 2012
What's this "sup2" you speak of? It's not on cisco's page!
 written by Pete Welcher, April 13, 2012 I agree, the linecard feature rules and other restrictions are a bit awkward. I can understand they're focusing their energies on efficient code development, but it means you really have to be up on what works and doesn't. I just heard of a case of a sale involving LRM's with N5K -- bzzzt, apparently still not supported. I used to feel this a bit with the 6500, but to a lesser degree. It's kind of "I just want to do good things with this box and linecards, I don't want to have to have a CCIE in it or spend a day making sure I've skimmed docs for gotchas, unexpected limitations, features, license minutiae, etc." I've been claiming for years that, in a sense, Cisco internal TME's (Tech Marketing Engineers) know too much. Good info, but it means they take this sort of complexity for granted, and maybe don't see the pain it causes customers. Or maybe they do but the Product Manager has different priorities or something. I did try to suggest some workarounds or at least designs to avoid in the blog.
written by Pete Welcher, April 13, 2012 Sssh, we're not supposed to be talking about Sup2! (Smile) Sup2 is a new supervisor for N7K, due out soon, not sure if it's been formally announced yet. I believe it is now orderable. Cisco SE's have been telling customers that are buying hardware to arrive in the June-July timeframe about it to avoid having to swap a shipped older Sup1 for a Sup2. Of course, new hardware & software = some risk.
 written by James, August 04, 2012
So this is what I ran into.
2 x Nexus 7K's in a Server Distribution role - L3 gateways for all the Currently 2 x M132 cards installed per chassis VPC Peer link split over the cards in a single port channel Single VDC. Multiple 5k's connected via dual side vpc portchannels 3 x 2k's per 7K, not dual attached, servers connected via vpc host port channels over the 2k's Known: F2 and M1, Won't operate in same VDC not willing to create secondary VDC Require the additional ports. SE's confirmed M1 & F2 in separate chassis with VPC would be fine** Small outages acceptable - under 5 mins Action (high level): Shutdown one side of the pair at the time. Install new cards Adjust config Bring up links Take down other side once hsrp/spanning tree/vpc synced and established. Result: Minimal Traffic interruption - Sub 30 seconds or so Problem: F2 line card in N7K-B, can not create VPC Peer Link with M1 card in N7K-A N7K-B, VPC Keep alive - active, VPC Role: None Configured (no election) No possible way to bring up VPC with current software release Unknown what would happen when N7K-B is in "none configured" role and N7K-A is shutdown If it was secondary, then split brain would happen after 3 missed keep alives, which would be acceptable as N7K-A would be shutdown for the same upgrade. Unknown if N7K-B would require reload to recover into a VPC primary role to take over? What happens if N7K-B comes up after a reboot, with no VPC keep alive, or VPC peerlink.. Would it be primary/active? Summary: Doesn't appear to be a stable / quick migration path to replacing M1 modules, with F2 modules in a VPC environment without a complete outage Issue could be resolved by software update to allow VPC to be established over Mixed cards.. Even if not supported.. Final Topology would not be M1-F2, but F2-F2. Am I missing something here, or is it just not possible to achieve this. Monday morning I'll be lining up a trip to Cisco to arrange a lab environment to test some of the above scenarios.
written by Pete Welcher, August 05, 2012 Thanks from me and readers for posting this information. Cross-card vPC is likely to be a problem. I think the intended approach is probably to create a separate VDC, put new cards one both sides, stand up the port channel and peer-link, then get rid of the old one. That does not exactly help with ports and member links that are in the old vPC however. You can't pre-migrate the port configs without allocating the ports. You can however re-allocate the ports to the new VDC and then paste in the saved config from the current VDC. You'd have to plan for uplinks and other details, but at least you wouldn't have to move cables, which would be rather time-consuming. US 2012 Networkers talk BRKDCT-2202 somewhat covers M1 to F1 vPC migration (to support FabricPath, which requires F-series card). Nice planning info and outage timing info. But that doesn't help with F2, where I'd think separate VDC would be desired.
 written by aleksandr, October 20, 2012
first great article
i do have a question though regarding the F2 series card when it comes to Inter Vlan routing the release notes on 6.1 state •Some software features are not available on the F2-Series module in Cisco NX-OS Release 6.0. These include ACL Capture, ERSPAN, FCoE, GRE tunnels, LISP, MACSEC, MPLS, NetFlow, online diagnostics, OTV, PIM-BiDir, and SVI or VLAN counters. Many of these features will be supported in future software releases; however, the F2-Series hardware does not support LISP, MPLS, and OTV, GRE tunnels, and PIM-BiDir so these features will remain unavailable on the F2-Series module. are SVIs actually supported on the F2 desing: Single Core with F2 48x10Gbps and Fexes - L2 to the edge, Nexus 7009 will perform inter-vlan routing, question is this possible with the F2 series ?
written by Pete Welcher, October 21, 2012
I always like being told it was a good article! Thanks.
 The F2 is supposed to be L2/L3 capable, which implies SVI capable to me. For instance, see http://www.cisco.com/en/US/pro...09336.html, which says they do L3. If you are concerned, talk to Cisco, maybe they can provide remote access to a N7K with F2 for you to do some test configuration on. Cisco has clearly stated that OTV, LISP, and MPLS ("advanced routing") will only be supported in the M1 and M2 cards. I will note they cost more, have larger routing table capabilities, and lower throughput than the F2 cards. Trade-off between features, price, and performance. In addition, the recent Cisco positioning is to clarify that the 6500 will have considerably more R&D and life and features that are aimed at campus functionality, whereas the Nexus line will receive only features that are deemed necessary for the datacenter. I take that as a rational allocation of effort and avoiding duplication of effort across hardware lines. To me, the historical precedent was the 7600/6500 split, where they tried to share hardware and ... let me diplomatically say I was never wild about the 7600. I heard from various sources how overloaded the engineers were for a while. It may be that this has a whiff of internal politics to some, i.e. "the 6500 empire fights back" to some. Turn that a bit though and it could also look like "the Nexus team needs to stay focused to maximize its ability to compete in a rapidly changing datacenter market". Hence I settle for "rational allocation of limited time and programming/engineering resources". Having said that, I cringe every time Cisco tries to tie features to what the market is perceived as, since I think some of the biggest successes have been devices used in ways the Cisco marketing side never anticipated. And other markets missed, e.g. the Metro Ethernet switches running a factor of 10 slower than the LAN switches for quite some time, and with that no small fast MPLS PE, in effect keeping MPLS restricted to the 6500 or those slower ME boxes. (Admittedly, cost, perceived market size, R&D cost, driving 6500 sales, other factors might have played into such a decision.) Another bit of the positioning I hear is that the 6500 will probably have some oversubscription, since that reduces cost but is not a major concern in most user-side campus environments. For that matter, WLAN 802.11ac isn't that far out, and should that be coupled with a faster way to get AP traffic onto the LAN, BYOD, WiFi-networked laptops, and WiFi-based printing could lead evolution towards diminished demand for a wired user campus. (Hmm, still might want wired phones? Or do things rapidly shift to bluetooth headsets driving softphones running on the BYOD cell phone or tablet?)
 written by James_, November 27, 2012
Hey Pete just a quick update, I ended up doing a proof of concept lab with Cisco to work out a valid migration strategy, and test their internal documentation vs. my split brain scenario.
Long story short, it is possible to migrate from a complete M1 environment to a F2 environment with less than 60 seconds worth of outage. The internal cisco document did require a new VDC and 2 x outages per vlan. If you want I can provide full testing details and the migration path. For the above poster, SVI's are most definitely supported on the F2 cards.
written by Pete Welcher, December 01, 2012 Thanks. Great to have actual lab data! A summary of the migration path might be of interest to readers. The 2012 U.S. Networkers had a talk about migrating to FabricPath. Buried in it was some coverage of getting a VPC peer link off M1 cards onto F1 cards, with a lot of discussion of outage times etc. Good stuff! I'd be interested in how the approach there compares to what you did. Seems like VDC or a 2nd chassis got added to the mix? Thanks.
written by James_, December 09, 2012
Summary:
Forced Splitbrain migration scenario to allow for minimal outage during a hardware upgrade from M1 based modules to F2 based modules. Alternative migration paths consisted of, a) Complete shutdown of server distribution switching – 15-30 minutes complete outage, B) Creation of temporary VDC, and one by one migration of HSRP, Spanning-tree, ports – 90 second outage PER vlan. Prep work: 1.Remove all QOS configs on existing M1 ports – QOS policy is not compatible with F2 modules. 2.FEX’s that are directly attached to the N7K will need to be taken offline for this Migration. 3.Need to ensure other switches are dual homed via vpc Part 1: Chassis B (current VPC secondary) 1.Enable Auto-Recovery on the VPC config 2.Shutdown VPC Peer link (shut Po1) (0.9 - 1.4 seconds outage over 50% of hosts) 3.Shutdown downstream links 4.Shutdown upstream links 5.Shutdown VPC Keep Alive (shut mgmt0) 6.Provision F2 cards a.vdc XXXXXX id 1 i.limit-resource module-type f2 ii.allocate interface Ethernet3/1-48, Ethernet 4/1-48 7.Apply port Config a.Refer port config spreadsheet (port mappings) 8.Recable Cards 9.Adjust spanning tree root to 20480 a.spanning-tree vlan xxx xxx xxxx priority 20480 b.spanning-tree vlan xxx xxx xxxx priority 20480 10.Ensure that Chassis B VPC status is “Secondary, Operational Primary” If not, then a reboot will be required on this chassis – 10 minutes to reboot, no outage 11.Bring up Upstream links 12.Bring up L3 link between chassis (0 - .4 second outage over 50% of hosts) Part 2 : Chassis A 1.Shutdown upstream links. 2.Shutdown downstream links *To be done simultaneously with Step 3* 3.Unshut downstream links on chassis B *To be done simultaneously with Step 2* 4.Wait for spanning tree to reconverge - approx 30 secs outage on inter-vlan and upstream traffic, intra-vlan traffic was unaffected. 5.Provision F2 cards a.vdc xxxxxxxx id 1 i.limit-resource module-type f2 ii.allocate interface Ethernet3/1-48, Ethernet 4/1-48 6.Apply port config a.Refer port config spreadsheet 7.Recable Chassis A 8.Bring up L3 links - no downtime 9.Bring up VPC peer link (no shut Po1 on chassis B) 10.Bring up VPC Keep alive link (no shut mgmt0 on chassis B) 11.Bring up downstream links – (0-1 second outage – This was variable, 2 tests we saw a brief traffic blip, 8 others we saw none) Test topology: 2 x Nexus 7010, 2 x Nexus 5548s, 2 x Catalyst 6500, Spirit test center, 2 x 2248, 2 x 2232 FEX’s Spirent test centre emulated 5000 MAC addresses over multiple vlans. Spirent test centre emulated both trunk and access port based hosts. All devices were running same NXOS revision as currently in Production Note: Main outage of 30 seconds attributed to spanning tree convergence / take over. An additional 30 second outage would be required to return the spanning-tree load balancing to the former state. This can be achieved by omitting step 9 within Part 1 The test procedure was tested approx 12 times, with consistent results obtained over the last 10, after the initial 2 tests finalised what config and the order of operations required. No reboots were required after the initial testing confirmed auto-recovery would enable split brain mode. Split brain mode consists of having both nexus 7010 distribution switches being in an active/active state, but only one side having active downstream links at any one time. Apologies, the formatting didn't copy/paste too well. Performed the actual change this weekend, apart from an isolated host experiencing an out of 3 minutes (high level issue) we had no issues, and the outage was approx 30 seconds. In short, No additional VDCs, no per vlan swapping, etc.
written by Pete Welcher, December 10, 2012 Nice details, interesting idea / approach. Leverage vPC behavior to force traffic to one side while you shift the other! With some planning and care (so the cables get shifted right), that keeps things (mostly) up and running.
 written by Roman, February 18, 2013
Greetings,
I am contemplating suitability of the F2(e) cards in a L3 DC core. After doing some additional research I have found that Cisco has deviated from their standard 8 queue design used on 6500 series and 7K M series linecards to a 4 queue design. Additionally they appeared to restrict the per port buffer size to approx 1.5MB I find quite anemic esp when compared to 65K or M series cards. What are your thoughts regarding this limitations and impact on deployment as L3 Core? Thank You for your very informative articles.
written by Pete Welcher, February 19, 2013
Interesting question, Roman. I pulled up the F2 and M2 datasheets.
The M2 shows 5.21 MB per port. The F2 shows 72 MB per module or 72/48 = 1.5 MB per port VOQ buffers. That is indeed less. I have a couple of thoughts about this: (1) You need less input buffering if you transfer data to other line cards at wire rate, which the F2 is designed to do. The M2 does not. It does up to 120 Mpps of L2 / L3 IPv4, half that of IPv6. (Unknown: what frame size was used for this.) The quick and dirty rule of thumb is that's about 120 Gbps or 12 x 10 Gbps or so of throughput, assuming the measurement was for the worst case, 64 B packets, and figuring an average packet size around 512 B. That may be a little generous, at 256 B average, you get half that, or if the 120 Mpps was for larger packets. Since there are 24 ports, that's 2:1 or so oversubscribed. That's probably why they didn't put 48 ports on it. In general, you need more queues and more buffering when there is oversubscription or transmission onto slower media. (2) I'd ask a different question: what are you planning to do with the linecard?. The F2 is intended for L2 and some L3, especially for features like FEX, FCoE, vPC, and FabricPath. You need the M1/M2 for things like OTV, LISP, MPLS VPN, and probably other WAN / L3 technologies. Which are you intending the primary use to be? (3) Following up on the L3 factor, note the M2 XL does up to 1 M IPv4 routes (approximately), 350 K IPv6 (and probably less of each if you're doing both -- they don't specify). In contrast, the F2 supports up to about 32,768 IPv4 routes and 16,384 IPv6. In a big network, the latter are fine if your datacenters are somewhat stubby as to routing. If you're like one site I was at a while back, running BGP in datacenter core to carry over 80,000 partner routes, that wouldn't work so well. (Of course, one might also claim having 80,000 routes and BGP in a datacenter core is a design problem as well.) (4) Everyone tends to assume more buffers are better. I sometimes think it must be a guy thing. That's not necessarily true! Google "bufferbloat" or see some of Terry Slattery's and my prior blogs about it. By another quick and dirty calculation, 1.5 MB x 8 b/B / 10,000 Mbps = 0.0012 seconds = 1.2 msec, that's not going to delay things by much if the buffer fills. Compare 5.21 MB: at 4.2 msec if the buffer fills. Still not too bad. If that creeps up to say 200 msec (as we've seen at one site where they "tuned" the buffers on a 6500), when there's congestion, TCP acts like you're sending packets a good distance and packet, generally slamming throughput. It gets worse if you drop packets occasionally. The irony here is you don't want to drop packets, which makes people want large buffers -- but hanging on to them in a buffer to the point of creating latency may actually be worse. F2 line card: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-719524.html M2 line card: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-706775.html
written by Roman, February 22, 2013
Thank you kindly for your very informative response.
As to my intended use: we are doing design on a budget so F2e appears more attractive option. It would be the only module in the 7K providing L3 "north" 10G connectivity out of DC to network core (10G) and providing L2 "south" 10G connectivity towards 5k/2k. Number of routes passes the muster (less than 6k) and no mpls (we use VRF-lite with some requirements such us PBR per VRF). Your buffering explanation was quite exhaustive (thank you for a quick lesson). The only thing you have not commented on is the difference in queueing architecture with the M series. I assume the deviation has to do with "intended use". In the ideal work one would deploy both the M series and F2e series however we are have to make a silk purse out of a sow's ear...
written by Pete Welcher, February 22, 2013 I didn't have much to say about the queuing. If you are at wire rate, you're not using the buffers much, and 4 vs. 8 queues is unlikely to make any difference. You might want to separate out voice, video, and FCoE, but anything else, probably not?
Write comment
|
I just heard something you might want to know about the Cisco Nexus 7000 F2 Card. That got me motivated to post some thoughts and questions I have about this interesting new linecard for the Nexus 7000. To cut to the chase, here's the key infonugget, something I heard from a savvy Cisco source: the FCoE functionality for the F2 card will require a Sup2 and licensing. 
So maybe in such a case you trade-in or sell off your F1 cards and replace them with F2 cards, if you conclude you need the performance. Or buy a new replacement chassis, and shift the M1/F1 chassis to a less-demanding role?
Subscribe to this comment's feed
I ran into the bottleneck issue you describe above just a couple of weeks ago. IMHO it's "gotchas" like that (and lots of others) that are holding the Nexus line back from being a runaway hit. Lots of good features but it seems like every time you put 3 or more together you run up against some limitation of the platform.