is this hardware good enough to build an OPNsense 2.5g router?

[toast0]

Don't buy new. Get used enterprise gear from ebay. https://www.ebay.com/itm/155513202027 is $13.50 and should work great, I have one and a half of this card. You'll need to run the silicom software once to set it up so it's never in bypass mode and you wouldn't know it wasn't a regular intel quad nic. You can get used dual 10g-base-T nics for not much more, although 10G switched are still expensive.

Edit: found a cheaper one $18 -> $13.50

 

[amd7674]

Don't buy new. Get used enterprise gear from ebay. https://www.ebay.com/itm/155513202027 is $13.50 and should work great, I have one and a half of this card. You'll need to run the silicom software once to set it up so it's never in bypass mode and you wouldn't know it wasn't a regular intel quad nic. You can get used dual 10g-base-T nics for not much more, although 10G switched are still expensive.

Edit: found a cheaper one $18 -> $13.50

Thank you very much

Do I need special silicom software when installing opnsense?

 

[SamirD]

You can get used dual 10g-base-T nics for not much more, although 10G switched are still expensive.

Ooooo...now I'm tempted to do an unneeded upgrade! Link?

 

[toast0]

Thank you very much

Do I need special silicom software when installing opnsense?

We discussed a bit in PMs, summarizing for everyone else: it should work on opnsense, but might be easier to setup on a different OS, OTOH, silicom's download site for drivers is down, so it's probably a bit too much fuss.

Everybody seems to want about $15-$20 to ship a network card to Canada. If you can wait a week, maybe I can ship you a card I have for shipping costs, and then I can replace it with something that has free shipping? My card will be all setup, so you don't need to run the silicom software; it'll just be a 4 port Intel gigE card that looks a little weird, but runs fine. I can even double check it shows up in OPNsense, so there's no surprises.

For SamirD, here's a regular dual 10G for $28 https://www.ebay.com/itm/155991474188 or this one for $16, but the ports are on the inside (also low profile bracket) https://www.ebay.com/itm/325887029837 this one for $25 also low profile https://www.ebay.com/itm/125516420586 or this one for $25 low profile and bypass (needs special software) https://www.ebay.com/itm/325891890948

A lot of these are best offer, so you can also try offering less and see what happens. If you want 10g SFP+, I saw a lot of dual port cards starting at $14 like this one https://www.ebay.com/itm/295632925317 (Emulex?) and this one at $15 https://www.ebay.com/itm/196029388733 (Intel)

 

[SamirD]

WOW! I had no idea 10GBase-T had dropped so low in the used market! This is absolutely fantastic for point to point connections over regular ethernet without a dac, and the SFP+ are even cheaper for where a DAC might be cheaper because I know where to get those pretty cheap. Thank you!

OP, you might want to consider 10Gb because the prices on multi-gig switches which support both 10Gb and 2.5Gb are falling, and can be found used or at the right deal for ~$100. That means for just maybe a little bit more on the switch (or not), and around the same price (or less) on the nics, you can actually have 10Gb vs just 2.5Gb. Some food for thought. Because while you're upgrading, you might as well upgrade.

 

[toast0]

Here's some options for the switches I ended up with for a little bit of 10g. 24 ports at 1g, 2x 10g-base-t, 2x sfp+

https://www.ebay.com/itm/374968844440 $110 + $30 shipping, has a bunch

https://www.ebay.com/itm/145527683081 $100 + $20 shipping, has PoE on half the ports, but only has one for sale, front is corroded, but the jack pins look ok to me?

The fans on these guys are pretty loud, and they're not super power efficient (on device monitoring says 16w with both 10g-base-t ports active and a good number of 1g active), but they worked for what I need (well want really; I've got two network closets with cat5e between them, wanted 10g between the two and 10g to a server at each location; I might pick a sfp+ card and then I can do 10g-base-t to one of the drops, cause that'd be fun, not cause I can do anything meaningful with it, lol ). Boot up slow as molasses too.

 

[amd7674]

Hey guys,
While waiting for x540 nic to arrive. I have a question.

Currently I have:
Arris TG4482A / XB7 Modem in bridge mode 2.5gb port --> EERO 6 pro (gateway / router) --> 16 ports Cisco gigabit switch --> other 2 wired EERO 6 pro APs / other devices (including ~30 amazon alexa smart devices)

In theory if I add 2 nics to the open opensense router I’m building. A single port 2.5g i226 + dual port x540.
Will I be able to use 2.5g port as WAN with my ISPs 2.5g port (1.5gb service) .

Then use one port 1gb on x540 with EERO (bridge) port —> 16 ports Cisco gigabit switch --> other 2 wired EERO 6 pro APs / other devices (including ~30 amazon alexa smart devices).

Then second port on x540 10g to multi-gig switch 10g port -> 2 x PCs with 2.5g nics and 1 PC with 10g nic

I believe EERO even in a bridge mode recommends to have one pod In upstream behind router. If I do that by using single x540 port I will lose ability to connect future multi-gig switch supporting 2.5g / 10g speeds.

However I wonder if that’s even possible for my opnsense router to be able to handle so much speed switching? CPU? Ram?

My initial end plan had this configuration (thanks to SamirD )

Arris TG4482A / XB7 Modem in bridge mode 2.5gb port --> OPNSense box 2.5Gb/Nbase-T nic with dual 10gb NICs to the LAN --> multi-gig unmanaged switch --> 16 ports Cisco gigabit switch --> 3 wired EERO 6 pro (bridge mode / APs) / other devices.

I must look into how firewalla guys with eero APs are doing it.

 

[SamirD]

In one word--yes. Yep, you'll have 2.5Gb from the modem to your opnsense and then I believe you can assign the same lan to both ports on the x540 (someone correct me if I'm wrong). The only concern, which I really don't know is a concern or not, is if switching between the two segments through the card introduces a bottleneck.

 

[amd7674]

In one word--yes. Yep, you'll have 2.5Gb from the modem to your opnsense and then I believe you can assign the same lan to both ports on the x540 (someone correct me if I'm wrong). The only concern, which I really don't know is a concern or not, is if switching between the two segments through the card introduces a bottleneck.

After reading some Opnsense threads and if I understand it correctly bridging ports and mixing speeds and using Opnsense multiple nics as switch it is not recommended (very CPU intensive)I believe using switch after would be better solution. Therefore I might just use 1 port for lan.

 

[SamirD]

After reading some Opnsense threads and if I understand it correctly bridging ports and mixing speeds and using Opnsense multiple nics as switch it is not recommended (very CPU intensive)I believe using switch after would be better solution. Therefore I might just use 1 port for lan.

Great research! Yep, then you can just use one 10Gb port to a multi-gig switch. But actually then you only need a single 10Gb port, right?

 

[amd7674]

Great research! Yep, then you can just use one 10Gb port to a multi-gig switch. But actually then you only need a single 10Gb port, right?

Maybe I could use second 10g port as lagg if I get managed switch ?

 

[SamirD]

Maybe I could use second 10g port as lagg if I get managed switch ?

You could but you wouldn't really need 20Gb/40Gb from the router to the LAN would you? And at point, you can just get a 25Gb or 40Gb nic. (This rabbit hole is endless, haha!)

 

[amd7674]

I think this is what I'm trying to achieve, will it work that is another question LOL

 

[SamirD]

That looks fantastic! How did you draw that up?

Oh, and the network layout looks fantastic too. If you wanted a bit more bandwidth to each of the gigabit switches you could connect all 3 directly to the multi-gig since otherwise the 1Gb link between the multi-gig and the first 1Gb switch will be a 1Gb bottleneck. Probably wouldn't matter at all for practical purposes unless you had the laptop and the PC with the 1Gb nic both maxing out their respective 1Gb bandwidths.

 

[toast0]

Ok, all this talk about 10g and I went and bought myself a dual-port sfp+ card and a 2 pack of DACs. Then I can put my work computer on 10g-base-t, if the house wiring cooperates. Not sure that'll be at all useful, since I don't really do stuff between the work computer and the home servers, and my internet connection is closer to 100mbps, but all my other desktops have a gigE switch at their drop cause there's other stuff there too. But it'll be fun?

 

[amd7674]

That looks fantastic! How did you draw that up?

Oh, and the network layout looks fantastic too. If you wanted a bit more bandwidth to each of the gigabit switches you could connect all 3 directly to the multi-gig since otherwise the 1Gb link between the multi-gig and the first 1Gb switch will be a 1Gb bottleneck. Probably wouldn't matter at all for practical purposes unless you had the laptop and the PC with the 1Gb nic both maxing out their respective 1Gb bandwidths.

draw.io

Yes, my first drawing wasn't done properly and it doesn't reflect my actual LAN setup. All secondary gigabit switches are connected to the main switch and I'm planning to keep it that way. Big thanks for your idea of 3 stage upgrade.

Current Setup:

Keeping EERO APs topology happy (Addding OPNSense router, dual x540 and single i226 pcie nics). WAN/LAN still at 1gb

End result , EEROs might not work (Adding multi-gig switch unmanaged, but managed would probably give me more options if needed) WAN 2.5gb / LAN multi-gig.

 

[Dopamin3]

XB7 I believe is an all in one gateway (e.g. it's doing routing). So you have to make sure you have it in "bridge" mode or whatever the ISP calls it so you don't double NAT. Friends don't let friends double NAT.

 

[amd7674]

XB7 I believe is an all in one gateway (e.g. it's doing routing). So you have to make sure you have it in "bridge" mode or whatever the ISP calls it so you don't double NAT. Friends don't let friends double NAT.

Yes, Xb7 is/was always in bridge. Eero 6 pro is/was router and I’m planning to replace it by Opnsense bare metal build and use Eeros strictly as APs.

 

[toast0]

Everybody cool is in here, so FYI, I saw this on serve the home article comments today https://www.amazon.com/TEROW-2-5G-POE-Multi-Speed-Compatible/dp/B0CGDCBBV3 if you clip the coupon, it shows up as $50 in checkout; but not sure if you can get it sent to Canada, shows unavailable on amazon.ca

4x 2.5g-base-t 2x sfp+, POE. I'm definitely not ordering one. But I might consider ordering the one the article was about, if it gets cheaper; that one was 4x 2.5g, 1x 10g-base-t, 1x sfp+ for $120 on aliexpress with poe, $90 without. Could be neat for me --- in my basement, I could use a sfp+ to my switch, then 10g-base-t to a drop; and at a drop, use 10g-base-t for the uplink, and have 2.5g for lesser desktops and 10g for something with an sfp+ card; if this config were $50, I'd probably have one on the way... at least my gaming desktop and my main desktop do run network backups, and 10g/2.5g could help with that a little... no big deal cause it happens after I go to bed, but still, somewhat useful.

 

[amd7674]

Hey.. Just an update. I've been playing with OPNSense for a bit and I got a simple install going. Since I didn't know how things will go I used 120g laptop HDD instead of SSD I have (I will comeback to that).
The install was a breeze and by following few guides / YT I was able to get it up and running on isolated network (as suggested 10.0.0.1/24). I've bought cheap 2.5g intel i226 pcie card (works great) (WAN) and used 1g intel onboard nic (LAN). I'm still waiting for two x540 cards to arrive.
So far I was able to get DNS Block Lists, CrowdSec (LAN/WAN) and wireguard client All seems to be working by limited testing. I wonder if I should try Suricata or Zenarmor, howeve I'm worry if my system will be able to handle it.
Right now I'm planning to move from 120g laptop HDD (UFS) to 120g ssd drive (ZFS). ALso I think I have somewhere another 120 ssd drive, and perhaps I should do ZFS mirror. I already backup up config file and I would do fresh reinstall / re-import config file.

As for network topology, once I do some more testing (perhaps including x540 NIcs) I will have EERO #1 off my main switch and EERO #2 and EERO #3 off other switches. This hopefully will work and if successful I will look into buying 10g multi-gig switch.

 

[amd7674]

For the mutli-gig switch I might spend few more $$$ to buy something that will last me longer. I see some cheaper 10gb switches are not lasting long (ports are dying).

I'm looking hard at Netgear xs508m and Qnap M2108R-2C

Any thoughts?

 

[German Muscle]

id bypass netgear, we used to use them at work and after alot of issues with them we are replacing them. Port failures, weird issues with loopbacks and sudden factory resets.

 

[Dopamin3]

id bypass netgear, we used to use them at work and after alot of issues with them we are replacing them. Port failures, weird issues with loopbacks and sudden factory resets.

I have an entirely different experience with Netgear.

At one of my work locations there is a 48 port Netgear GS752TP that's been chugging for forever that is almost fully populated, 0 issues. At home my "core" switch is a Netgear MS510TXUP and one of my "access" switches is a Netgear GS110EMX. Have had them for a few years now with absolutely no issues.

For the mutli-gig switch I might spend few more $$$ to buy something that will last me longer. I see some cheaper 10gb switches are not lasting long (ports are dying).

I'm looking hard at Netgear xs508m and Qnap M2108R-2C

Any thoughts?

XS508M would be fine. There is also the TP-Link TL-SX1008 to consider. I don't know Canadian pricing but it's cheaper in USA than the XS508M. The least reputable and lesser known Hasivo S1100WP-8XGT-SE is by far the cheapest and a very small upcharge to get POE+ on the ports. I would avoid the QNAP personally.

 

[amd7674]

XS508M would be fine. There is also the TP-Link TL-SX1008 to consider. I don't know Canadian pricing but it's cheaper in USA than the XS508M. The least reputable and lesser known Hasivo S1100WP-8XGT-SE is by far the cheapest and a very small upcharge to get POE+ on the ports. I would avoid the QNAP personally.

Thank you. Yes, Sx1008 is a cheaper option, however there is a lot reports of dying ports within a year or so. Also I want to buy something from Amazon so I have 30 days to test it before I keep it.

 

[German Muscle]

I have an entirely different experience with Netgear.

At one of my work locations there is a 48 port Netgear GS752TP that's been chugging for forever that is almost fully populated, 0 issues. At home my "core" switch is a Netgear MS510TXUP and one of my "access" switches is a Netgear GS110EMX. Have had them for a few years now with absolutely no issues.

Are they just default configured? if so that is probably why. We had 3-4 at every site, over 40 sites. When you have to work all day then travel 2-8 hours to a site cause the switch went rogue then drive back 4 hours and then sleep and be in the next morning like 10 times then youd understand. We used the GS752s and the v2 model.

 

[Dopamin3]

Are they just default configured? if so that is probably why. We had 3-4 at every site, over 40 sites. When you have to work all day then travel 2-8 hours to a site cause the switch went rogue then drive back 4 hours and then sleep and be in the next morning like 10 times then youd understand. We used the GS752s and the v2 model.

At the work location with GS752TP there are some VLANs for guest WiFi and deskphones (both being powered by the POE), QoS and a static route or two. My MS510TXUP at home just basically has VLANs on some ports and only provides POE to two access points. Definitely more than just the default config but nothing too crazy.

I only have a sample size of one with each of these switches so they've worked for me and I had a friend in trade school type deal go full stack Netgear Business after leaving Cisco and that's what got me to try them at home. The GS752TP was already long deployed and I haven't had a reason to replace it yet. Most of the higher end Netgear switches come with "lifetime" protection but I've never had to try to RMA.

 

[amd7674]

My options are very limited because I want RJ45 10gb multi-gig switch.

• QNAP QSW-2104-2T-A-US / TRENDnet TEG-S762 <-- Cheapest only 6 ports ( 2 x 10gb and 4 2.5gb) and also a lot reviews with reliability problems.
• TP-Link TL-SX1008 is the cheapest 8 port but there are tons of user reviews it dies within a year, also it has very noisy fan which can be replaced by Noctua but you will lose warranty.
• Netgear XS508M

 

[Dopamin3]

My options are very limited because I want RJ45 10gb multi-gig switch.

• QNAP QSW-2104-2T-A-US / TRENDnet TEG-S762 <-- Cheapest only 6 ports ( 2 x 10gb and 4 2.5gb) and also a lot reviews with reliability problems.
• TP-Link TL-SX1008 is the cheapest 8 port but there are tons of user reviews it dies within a year, also it has very noisy fan which can be replaced by Noctua but you will lose warranty.
• Netgear XS508M

Hasivo S1100WP-8XGT-SE

There are enough people using this switch now thanks to the coverage from STH that one can decide if it's worth the "risk". Check AliExpress reviews, STH article/YouTube, other forums and YouTube in general. Setting a static IP on it seems like it takes a few steps but other than that paying <$300 for a brand new MANAGED 8 port 10GbE is fantastic. Adding POE only takes the cost up minimally too. I would also avoid the TL-SX1008 since it seems like failure rate is incredibly high.

I really want to get one to see what it's like first-hand but honestly I have no need for it right now. I've got 10GbE links already where I need them. One day I might replace my GS110EMX with it (which is a weird switch with only 2x 10GbE and 8x 1GbE) but where it's deployed it makes sense.

 

[amd7674]

Hasivo S1100WP-8XGT-SE

There are enough people using this switch now thanks to the coverage from STH that one can decide if it's worth the "risk". Check AliExpress reviews, STH article/YouTube, other forums and YouTube in general. Setting a static IP on it seems like it takes a few steps but other than that paying <$300 for a brand new MANAGED 8 port 10GbE is fantastic. Adding POE only takes the cost up minimally too. I would also avoid the TL-SX1008 since it seems like failure rate is incredibly high.

I really want to get one to see what it's like first-hand but honestly I have no need for it right now. I've got 10GbE links already where I need them. One day I might replace my GS110EMX with it (which is a weird switch with only 2x 10GbE and 8x 1GbE) but where it's deployed it makes sense.

thank you However as much as it is tempting I'm not risking spending money on something that may not work. I will most likely buy Netgear XS508M, local warranty and I can actually try it on my network for 30 days.

 

[amd7674]

Just an update. Everything seems to be working now. I got main EERO off, and a "waterfall" EERO setup seems to work.

Now I'm looking to buy cheap 10g mutli-gig router.... TP-Link TL-SX1008 is +$200 cheaper than Netgear XS508M (my option).

What do you guys think? TP-Link has a lot of premature port dying.... hmmmmm

My end result is this: