PVE management interface on virtualized NIC?

[Levent]

I am trying to figure out how I would go about moving PVE's management interface to a interface that I will be powering via VM inside the PVE. Idea behind this to avoid exposing management port to WAN and freeing up an ethernet port on the device.

 

Any ideas? Is it as simple as editing /etc/network/interfaces to the vmbr of my choice (alongside with the IP)? I got 3 other physical ports on this device and all I have assigned those to LAN in pfsense VM already.

[LIGISTX]

29 minutes ago, Levent said:

I am trying to figure out how I would go about moving PVE's management interface to a interface that I will be powering via VM inside the PVE. Idea behind this to avoid exposing management port to WAN and freeing up an ethernet port on the device.

 

Any ideas? Is it as simple as editing /etc/network/interfaces to the vmbr of my choice (alongside with the IP)? I got 3 other physical ports on this device and all I have assigned those to LAN in pfsense VM already.

Why would you need to do this to not expose it to WAN? 
 

If you want to run virtual pfsense, you should get a PCIe NIC and pass that through to pfsense so it has bare metal access to that NIC. Only use 2 ports on the NIC, one for WAN, one for LAN, LAN goes out to a switch. From the switch, all devices plug into it, including the PVE host itself (the port on the mobo that PVE is running on). 
 

Don't use pfsense as a switch, use a switch as a switch. 

[Levent]

1 minute ago, LIGISTX said:

Why would you need to do this to not expose it to WAN? 
 

If you want to run virtual pfsense, you should get a PCIe NIC and pass that through to pfsense so it has bare metal access to that NIC. Only use 2 ports on the NIC, one for WAN, one for LAN, LAN goes out to a switch. From the switch, all devices plug into it, including the PVE host itself (the port on the mobo that PVE is running on). 
 

Don't use pfsense as a switch, use a switch as a switch. 

I would much prefer have WAN assigned straight into pfsense VM. I also have physical console access to this device, so push comes to shove I can carry one of the 5 monitors I got downstairs and plug it in and fix the config. I also dont like having PVE exposed to internet due to security (I have fail2ban setup on all my devices in any case but still).

[LIGISTX]

28 minutes ago, Levent said:

I would much prefer have WAN assigned straight into pfsense VM. I also have physical console access to this device, so push comes to shove I can carry one of the 5 monitors I got downstairs and plug it in and fix the config. I also dont like having PVE exposed to internet due to security (I have fail2ban setup on all my devices in any case but still).

Right… what I am suggesting is exactly what you want… 

 

Put an Intel PCIe NIC in that machine (if you don’t already have one), pass it through to the pfsense VM so that VM is the only machine that can access it. 
 

Plug WAN into port 0 of that PCIe NIC, plug LAN into port 1. Now WAN goes to pfsense, and only pfsense. From the port 1 (LAN), go to a switch. Now at this point, your network is like it’s not virtualized at all. Plug all devices into that switch, including your proxmox host.

 

I run a virtual pfsense network, this is literally exactly what I do. PVE should NOT be exposed to the internet, ever. 

[Levent]

21 minutes ago, LIGISTX said:

Put an Intel PCIe NIC in that machine (if you don’t already have one), pass it through to the pfsense VM so that VM is the only machine that can access it. 

That is going to be a little problematic lol. Device in question is below.

I have lots of spare ram in this so I am planning on moving some of the light vms into this and saving up as much ram in my actual homeserver.

 

[LIGISTX]

54 minutes ago, Levent said:

That is going to be a little problematic lol. Device in question is below.

I have lots of spare ram in this so I am planning on moving some of the light vms into this and saving up as much ram in my actual homeserver.

 

Does whatever NIC solution that uses have a way to pass them through to VM’s? On my quad port Intel NIC I can pass them through to pfsense in groups of 2. If you could do that with this, you’d be able to do what I suggest. But if not…personally I wouldn’t run pfsense under proxmox on that.
 

The only proper way to do it with that hardware would be with a managed switch between modem and that box, and use vlans. People do this in homelab world, but I wouldn’t do that myself. can you just leave pfsense on your main homelab system?

[leadeater]

1 hour ago, Levent said:

That is going to be a little problematic lol. Device in question is below.

Do you have a VLAN capable switch or do the NICs in that computer support SR-IOV?

[Levent]

16 minutes ago, LIGISTX said:

Does whatever NIC solution that uses have a way to pass them through to VM’s? On my quad port Intel NIC I can pass them through to pfsense in groups of 2. If you could do that with this, you’d be able to do what I suggest. But if not…personally I wouldn’t run pfsense under proxmox on that.
 

The only proper way to do it with that hardware would be with a managed switch between modem and that box, and use vlans. People do this in homelab world, but I wouldn’t do that myself. can you just leave pfsense on your main homelab system?

I thought about that and I actually spent easily 10 minutes browsing this things many available bios options and sadly did not see IOMMU related option (I actually thought this would have it but I might be SOL). I dm’d the aliexpress seller about it 30 minutes ago, hopefully I can enable it.

3 minutes ago, leadeater said:

Do you have a VLAN capable switch or do the NICs in that computer support SR-IOV?

I sadly don’t have a managed switch. 

[Electronics Wizardy]

2 hours ago, Levent said:

I am trying to figure out how I would go about moving PVE's management interface to a interface that I will be powering via VM inside the PVE. Idea behind this to avoid exposing management port to WAN and freeing up an ethernet port on the device.

 

Any ideas? Is it as simple as editing /etc/network/interfaces to the vmbr of my choice (alongside with the IP)? I got 3 other physical ports on this device and all I have assigned those to LAN in pfsense VM already.

If you put a IP address on a bridge, the Proxmox host can connect to it, otherwise it can't.

 

I'd normally keep the current vmbr0 as your lan bridge, and let proxmox use that for the host. Then use the other bridges for wan and other things. 

 

If you wan't multiple lan ports, I'd connect them all to the bridge in Proxmox, not via the router. That will be easier to setup and manage.

[leadeater]

7 minutes ago, Levent said:

I sadly don’t have a managed switch. 

Check if you have SR-IOV then, with that you can make virtual physical interfaces then pass that through to a VM.

 

https://hackmd.io/@2kHYGtJaRV-DxVpSmefW0w/B1z-mTDEi

 

Edit:

Oh ignore

8 minutes ago, Levent said:

I thought about that and I actually spent easily 10 minutes browsing this things many available bios options and sadly did not see IOMMU related option (I actually thought this would have it but I might be SOL). I dm’d the aliexpress seller about it 30 minutes ago, hopefully I can enable it.

Based on that ^ no

[Levent]

3 hours ago, leadeater said:

Check if you have SR-IOV then, with that you can make virtual physical interfaces then pass that through to a VM.

 

https://hackmd.io/@2kHYGtJaRV-DxVpSmefW0w/B1z-mTDEi

 

Edit:

Oh ignore

Based on that ^ no

Oh well time has come to get a managed switch either way. I can help out the company by requesting to purchase of one of their rack mounted HP 1810-24G units (for like $20 lol) only if I can get a yay or nay vote here. Otherwise I am up for cheapo managed switch recommendations. I wouldnt need more than 5 to 10 1Gbit ports for this device.

[Levent]

Actually wait I am dumb, never assume something and always double check. It does support IOMMU.

fucks sake, are they all in the same group? am I tripping?

[LIGISTX]

36 minutes ago, Levent said:

Actually wait I am dumb, never assume something and always double check. It does support IOMMU.

fucks sake, are they all in the same group? am I tripping?

I think they are all individual? Go in proxmox and see what pops up when you go to add a PCIe device to a VM…

[Levent]

23 minutes ago, LIGISTX said:

I think they are all individual? Go in proxmox and see what pops up when you go to add a PCIe device to a VM…

I am confused more than before now, lol.

[LIGISTX]

7 minutes ago, Levent said:

I am confused more than before now, lol.

Did you enable it in proxmox kernel?

 

https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/

 

Specifically step 3a.

[Levent]

Just now, LIGISTX said:

Did you enable it in proxmox kernel?

 

https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/

Actually I *just* did that 15 seconds ago and voila!

 

Going back to the original problem,

 

ETH0 (WAN) > Currently assigned to vmbr0 and pve management interface runs on this port. I dont want that and I also dont want pve to be exposed to wan. I want to pass this NIC straight into pfsense.

ETH1 (LAN) > Will be used to connect to an AP downstairs.

ETH2 (LAN) > Will be used to server, AP and other wired clients upstairs.

ETH3 (LAN) > Will be used for connecting a wired machine here and there.

 

I still dont follow how I would go about accessing PVE once I passed all interfaces to pfsense.

[LIGISTX]

13 minutes ago, Levent said:

Actually I *just* did that 15 seconds ago and voila!

 

Going back to the original problem,

 

ETH0 (WAN) > Currently assigned to vmbr0 and pve management interface runs on this port. I dont want that and I also dont want pve to be exposed to wan. I want to pass this NIC straight into pfsense.

ETH1 (LAN) > Will be used to connect to an AP downstairs.

ETH2 (LAN) > Will be used to server, AP and other wired clients upstairs.

ETH3 (LAN) > Will be used for connecting a wired machine here and there.

 

I still dont follow how I would go about accessing PVE once I passed all interfaces to pfsense.

You don’t pass all interfaces to pfsense. Pass only 2, 1 for WAN, 1 for LAN.  Don’t use pfsense as a switch…. Bridging NIC’s just doesn’t make much sense when you can buy a 5 port gigabit switch for like 11 dollars.

 

What I would do simply to remove potential wiring issues, I’d plug WAN into eth0, LAN into eth1, a dummy plug into eth2, and then in /etc/network/interfaces you would assign the Proxmox bridge port to eth3. That’ll put proxmox on eth3. 
 

Plug eth 1 (pfsense LAN) into a switch, and plug EVERYTHING into that switch; your AP’s, your other switches, proxmox eth3, your other servers, everything. 

[Electronics Wizardy]

41 minutes ago, Levent said:

Actually I *just* did that 15 seconds ago and voila!

 

Going back to the original problem,

 

ETH0 (WAN) > Currently assigned to vmbr0 and pve management interface runs on this port. I dont want that and I also dont want pve to be exposed to wan. I want to pass this NIC straight into pfsense.

ETH1 (LAN) > Will be used to connect to an AP downstairs.

ETH2 (LAN) > Will be used to server, AP and other wired clients upstairs.

ETH3 (LAN) > Will be used for connecting a wired machine here and there.

 

I still dont follow how I would go about accessing PVE once I passed all interfaces to pfsense.

I'd skip pcie passthrough here. 

 

Just give your router vm 2 nics, one for lan and one for wan. Then you can hve it so eth1,2,3 are all connected to vmbr0 nd the proxmox mngaement. Then create vmbr1 and connect that to eth0 and the wan port on the vm.

 

 

[Levent]

Quick update, figured it out. It was as I thought in the OP.

 

I created 5 vmbrs.

vmbr0=bonded to eth0 (which will be used for WAN)

vmbr1=bonded to eth1 (which will be used for LAN)

vmbr2=bonded to eth2 (which will be used for LAN)

vmbr3=bonded to eth3 (which will be used for LAN)

vmbr4=no binds.

 

1. Set up interfaces on the pfsense console
   1. Assigned WAN to a random interface and setup other vmbridges properly
   2. Setup LAN and enabled DHCP on other vmbrs that are going to be LAN ports.
2. I then edited etc/networks/interfaces to have vmbr4 a static IP (10.10.0.2) I also removed the network adapter bond proxmox creates by default so that default network adapter (in my case was eth0) would be freed up. I also edited hosts file to reflect the IP change I made above.
3. I then plugged my laptop to one of the LAN ports physically then setup port forwarding to 10.10.0.2:8006
4. It works! I gotta forward couple of more ports but it at least works.

I now gotta figure out how I would go about assigning different vmbrs to same interface/LAN. Any ideas? I would much rather not spend any money at all and also save up on precious space.

[Electronics Wizardy]

1 hour ago, Levent said:

Quick update, figured it out. It was as I thought in the OP.

 

I created 5 vmbrs.

vmbr0=bonded to eth0 (which will be used for WAN)

vmbr1=bonded to eth1 (which will be used for LAN)

vmbr2=bonded to eth2 (which will be used for LAN)

vmbr3=bonded to eth3 (which will be used for LAN)

vmbr4=no binds.

 

1. Set up interfaces on the pfsense console
   1. Assigned WAN to a random interface and setup other vmbridges properly
   2. Setup LAN and enabled DHCP on other vmbrs that are going to be LAN ports.
2. I then edited etc/networks/interfaces to have vmbr4 a static IP (10.10.0.2) I also removed the network adapter bond proxmox creates by default so that default network adapter (in my case was eth0) would be freed up. I also edited hosts file to reflect the IP change I made above.
3. I then plugged my laptop to one of the LAN ports physically then setup port forwarding to 10.10.0.2:8006
4. It works! I gotta forward couple of more ports but it at least works.

I now gotta figure out how I would go about assigning different vmbrs to same interface/LAN. Any ideas? I would much rather not spend any money at all and also save up on precious space.

Are all your lan interfaces bridged? If they are, make it so there is one lan bride that uses all the 3 physical lan ports. That should make it easier to setup and manage here. Then get rid of vmbr2 and vmbr3 , and just connect those ethernet ports to vmbr1.

[Levent]

8 minutes ago, Electronics Wizardy said:

Are all your lan interfaces bridged? If they are, make it so there is one lan bride that uses all the 3 physical lan ports. That should make it easier to setup and manage here. Then get rid of vmbr2 and vmbr3 , and just connect those ethernet ports to vmbr1.

holy crap I bridged them in pfsense but never thought of bridging them in proxmox itself. That makes VLANs whole lot more easier for me too!. Thanks for the tip!

[Electronics Wizardy]

1 hour ago, Levent said:

holy crap I bridged them in pfsense but never thought of bridging them in proxmox itself. That makes VLANs whole lot more easier for me too!. Thanks for the tip!

Also for vlans, I'd set it up with multiple virtual brides in proxmox, assigned to a vlan. That way proxmox handles all the vlans, not pfsense, and you just add more virtual nics to pfsense .

[Levent]

Any ideas on what would cause VLANs to work for internally but not through physical ethernet ports?

 

I simplified the setup quite a bit. I have WAN bridge set to enp2s0, LAN bridge set to enp3s0 enp4s0 and enp5s0. I have vlan tagging enabled on LAN bridge and I have added lan bridge multiple times as different NICs to pfsense using different as different VLAN Tags.

 

for the arguments sake:

vtnet0: wan (out of this arguments context)

vtnet1: lan (VLAN 50)

vtnet2 lan2 (VLAN 100)

vtnet3 guest (VLAN 200)

I got DHCP server running vtnet1,2 and 3 I also confirmed VM created within proxmox is able to get IP and connect to the networks. However when I connect my laptop to any of the enp2-3-4s0 ports I am not able to connect to any vlans (I even tested on windows and on Linux Mint). Am I missing something?

[leadeater]

10 hours ago, Levent said:

I got DHCP server running vtnet1,2 and 3 I also confirmed VM created within proxmox is able to get IP and connect to the networks. However when I connect my laptop to any of the enp2-3-4s0 ports I am not able to connect to any vlans (I even tested on windows and on Linux Mint). Am I missing something?

Your laptop NIC would need to be configured for each VLAN to work, Proxmox is handling the tagging and untagging internally but once it goes out a physical port it'll have the associated VLAN tag on the Ethernet frame so unless you have a managed switch or setup that laptop with one or more VLAN interfaces it's only going to accept and send untagged Ethernet frames.

 

At least I believe that is the situation here.

 

Set-NetAdapter –Name "Ethernet0" -VlanID 50

 

Some NICs support creating multiple VLAN sub interfaces in the hardware settings of the network adapter, not user if desktop OS supports it or not, seen a mix of yes and no 

[Levent]

1 hour ago, leadeater said:

Your laptop NIC would need to be configured for each VLAN to work, Proxmox is handling the tagging and untagging internally but once it goes out a physical port it'll have the associated VLAN tag on the Ethernet frame so unless you have a managed switch or setup that laptop with one or more VLAN interfaces it's only going to accept and send untagged Ethernet frames.

 

At least I believe that is the situation here.

 

Set-NetAdapter –Name "Ethernet0" -VlanID 50

 

Some NICs support creating multiple VLAN sub interfaces in the hardware settings of the network adapter, not user if desktop OS supports it or not, seen a mix of yes and no 

Actually issue turned out to be problem on the proxmox side. Even if you create Linux Vlans through interface, it doesnt add "vlan-id xx" to the /etc/networks/interfaces. I have done that manually and so far *knocks his head into his desk* everything seems to be working. Thanks!

 

Edit: So what I have do so far is (so that I can come back here when I inevitably lock myself out of my own network lol):

1. create a linux bridge with all the physical ports attached and Vlan aware tag clicked on the bridge menu (this is now vmbr1).
2. Then I created vlans in proxmox networks using the same VLAN numbers I planned on using (vlan50, vlan100, vlan200).
3. I added "vlan-id 50", "vlan-id 100", "vlan-id 200" to each vlan in proxmox interfaces manually.
4. I restarted networking on the proxmox gui.
5. I assigned the physical ports bridge (vmbr1) to pfsense then I also created vlans in pfsense using the same numbers above.
6. I then proceeded to setup networks using the VLANs I created in pfsense.
7. I havent assigned the actual vmbr1 to anything in pfsense.