Planning for firewall device vs running VMs

[BeastChan23]

Hey folks, I'm planning on switching off my ISP provided modem/router due to it being pretty limiting in port forwarding, and I suspect it isn't allowing my friends to connect to a game server I was attempting to host.
Hosting a server has worked in the past, however I think at the time I was using my own modem, along with my own Asus router.
At some point they decided I need their router for unlimted data, but my old modem should still work last time I checked.
I currently have over 1 gig service, so I've been forced to use their modem since it has a 2.5GbE port. I've already invested in a multi gig switch with 1/2.5GbE and SFP+ ports, I've been using this with the provided modem/router just fine.

Current issue is really just that modem that doesn't allow me to adjust firewall settings, logging into the modem forwards me to the ISP site to remote control it, but there is no port forwarding options at all.

To resolve this I've come across two possible solutions:
1. Use an existing filer server I'm using and repurpose it from just serving files, and running plex to using it with Proxmox to do the same thing and also run opnSense to function as firewall for network
2. Purchase an Intel based box from aliexpress with 4x 2.5GbE ports barebone.

Buying a NIC with multiple ports alone would be around $50, buying the box plus storage and ram would be over $200.
Not having to redo my OS on my file server would be nice, and I'm not sure if I'd need to upgrade my CPU and memory for running at least two VMs, I currently have my file server running of 16GBs of ram with a Ryzen 5600G and GTX 1650 Super

Does anyone have any other alternative solutions or should I just go with the multiport box and not mess with my current server setup?

[Levent]

24 minutes ago, BeastChan23 said:

I'm planning on switching off my ISP provided modem/router due to it being pretty limiting in port forwarding, and I suspect it isn't allowing my friends to connect to a game server I was attempting to host.

Maybe it is because your ISP provides you a CGNAT connection (very common nowadays). Check the IP your modem gets and compare it with your actual ip (by googling what is my ip), if those dont match then there is no reason to upgrade your setup.

[wseaton]

Also double check with your ISP you can run your own gear. Many dont allow this on gig links. 

 

And...its possible they are blocking the port forwarding. Some are dicks about hosting your own stuff.

[BeastChan23]

I did use my pre existing Motorola modem with my 1 gig connection before, the whole reason I made sure to get a docsis 3.1 modem, Comcast just decided that their modem is required for their unlimited data option. 

 

I'll double check and compare the ip address provided

 

Quick search claims that Comcast doesn't do cgnat, but again I'll check when I get back home

[BeastChan23]

16 hours ago, Levent said:

Maybe it is because your ISP provides you a CGNAT connection (very common nowadays). Check the IP your modem gets and compare it with your actual ip (by googling what is my ip), if those dont match then there is no reason to upgrade your setup.

Confirmed IP addresses match

[Bdavis]

check out the articles on the serve the home website, they have recently done some reviews on some 2.5gb boxes that are capable of running pfsense. I don't really recommend virtualizing your router/firewall for production environments. My personal setup is a hp thin client with a dual 2.5gb NIC running pfsense.