Tiernan's Comms Closet

Day 14 of #100daysofhomelab and I have been thinking about future upgrades if I had the money… So, I have my CloudShed in the back garden. Currently, I only have an HP Micro Server and a (not currently in production) Dell R720, along with a Ubiquiti Edge Switch 48 Lite. Between the Shed and the house is a fibre link purchased through FS.com, with 6 pairs. Currently, only 1 pair is in use, giving me a 10Gb/s between the house and shed, and with the easy option to upgrade to 20Gb. But I have been thinking bigger.

I have been looking at the Mikrotik CCR2004-1G-2XS-PCIe (a bit of a mouthful…) SmartNIC. It’s a full MikroTik router on a PCIe Card. It has a Quad-core ARM Processor, 4GB RAM, some storage and 2X25Gbit/s Interfaces… Well, technically, 4… there are 2 front connectors and 4 that the host server sees… If I am reading the diagram below correctly, it looks like all ports are seen by the Host, but 2 go through the bridge and 2 go direct… I haven’t played with one yet (Mikrotik, if you are listening, hint, hint!) so not sure how it would work… A review from Alyx Wijers says that on the Linux box they tried, the 10Gb SFP+ module they had shows in passthrough and the other 2 are connected to the bridge… Ideally, for the ideas i have for this, I passthrough would be handy for stuff like storage, but i would want the rest of my traffic going over that bridge interface… Or at least i think thats how it would work…

The card has 2X25Gb ports (SFP28 ports) that connect to the rest of your network, for example to a CRS504-4XQ-IN switch (4x100Gb ports, which can be broken out into 4x25Gb ports each…). If you go through the bridge, you get all the features of RouterOS, like firewall rules, VXLAN, etc, all in the NIC. The switch then doesn’t need to do as much, letting it do the switching and leaving everything else at a NIC level. If you use passthrough, you, essentially, bypass the router/firewall rules… I think…

So, what would my plan for the upgrade be? Well, this is where things get expensive… I would need 2 of the switches (one in the house, one in the shed linked with a single 100Gb fibre). Then, I would need 8 of the SmartNICs (GodboxV3, GodBoxV2, 1 for each of the R720s (second one coming soon), 1 for each of the R620s (coming soon), one for the HP DL380 G8 (also pending) and 1 for the big storage box… again, pending). The plan would be that GodBoxV2 and V3 would be in the house, and both connect to the house switch at 25Gb a sec. there would be uplinks to internal 10Gb switches along with the UDM Pro.

In the shed, the 6 servers would each connect to the switch at 25Gb, using 10 of the renaming 12 ports. The R720s, Storage Box and HP will probably get 2x25Gb connections. In theory, the R620s could also connect at 50Gb but I would have no extra room later… Might not be a major issue, mind you. There would be spare ports in the house… I could, in theory, get a second 100Gb switch for the shed! 😛

But, what would this cost? Well, current prices are showing that the cards are around 200EUR a pop and the switch is just shy of 800 quid… so, for a little under €3200, I could get 2 switches and 8 NICs. I would need break-out cables, 100Gb Optics, and some other bits, so, say 500 quid for that… So, just under 4k? One of these days, hey! I can dream!

MikroTik CRS504-4XQ-IN Review Momentus 4x 100GbE and 25GbE Desktop Switch (servethehome.com)

Review: A Dive into Mikrotik’s Weird SmartNIC (CCR2004-1G-2XS-PCIe) // Alyx Wijers