significantly slower Wireguard speeds after moving pfSense to new hardware

[Mnky313]

So I recently moved pfSense to a new system (2x xeon e5-2620, 16gb RAM) (fresh install, not backup/restore) and wireguard speeds are significantly slower than both running the wireguard app on windows and connecting to the same server & the old system that had pfSense on it (fx-7600p, 4gb RAM).

I'm connecting to the same server on all tests:

without a vpn:

924mbps down
511mbps up

running wireguard on my windows pc/(speeds are similar on pfSense on the old hardware), this also flucuates between ~400-700mbps down most of the time:
564mbps down

133mbps up

 

running wireguard on pfSense:

158mbps down

196mbps up

 

I've also tried different servers and ports and that doesn't seem to make any difference.
(& MTU/MSS is set to 1420, tried lowering it to 1280 and that also didn't make a difference.)

[Alex Atkin UK]

51 minutes ago, Mnky313 said:

So I recently moved pfSense to a new system (2x xeon e5-2620, 16gb RAM) (fresh install, not backup/restore) and wireguard speeds are significantly slower than both running the wireguard app on windows and connecting to the same server & the old system that had pfSense on it (fx-7600p, 4gb RAM).

I'm connecting to the same server on all tests:

without a vpn:

924mbps down
511mbps up

running wireguard on my windows pc/(speeds are similar on pfSense on the old hardware), this also flucuates between ~400-700mbps down most of the time:
564mbps down

133mbps up

 

running wireguard on pfSense:

158mbps down

196mbps up

 

I've also tried different servers and ports and that doesn't seem to make any difference.
(& MTU/MSS is set to 1420, tried lowering it to 1280 and that also didn't make a difference.)

Doesn't explain why the difference is so big, but the e5-2620 seems to have less single-thread performance which could be impacting this.

 

Given you've moved to 2x Xeon, is it running in a VM?  That could also have a performance hit if so.

[LIGISTX]

20 minutes ago, Alex Atkin UK said:

Doesn't explain why the difference is so big, but the e5-2620 seems to have less single-thread performance which could be impacting this.

 

Given you've moved to 2x Xeon, is it running in a VM?  That could also have a performance hit if so.

To elaborate on this, if it is in a VM, what hypervisor are you using? I know with proxmox I had to edit the CPU to not emulate, but to actually be “system” which allowed me to use hardware accelerated encryption/decryption within pfsense. 

[Mnky313]

29 minutes ago, Alex Atkin UK said:

Doesn't explain why the difference is so big, but the e5-2620 seems to have less single-thread performance which could be impacting this.

 

Given you've moved to 2x Xeon, is it running in a VM?  That could also have a performance hit if so.

It is not in a VM, I noticed that there was a small drop in single threaded performance (there's 2 fx-7600p on passmark and the one with more results is ~1200pts) but it seems like it should be fine (they tested an intel Atom and were getting ~900mbps)

 

If it is CPU related, 2667v2 / 4627v2 are pretty cheap.

I'm also not getting near 10Gbe speeds on 2 bridged interfaces (~2.7Gbps) but I was warned that bridging interfaces isn't great on pfSense so that could be unrelated (the bridges interfaces are seperate from this stuff).

I wonder if it is just a cpu limitation, is there a was to check per core load while running a speedtest or something to see if that's the issue?

[Needfuldoer]

What platform are you running?

 

Can you step it up to a pair of faster processors, like two E5-2667 v2s?

 

Depending on your firewall needs, you might be better off with a Haswell or Skylake era desktop with a couple extra NICs thrown into it. (That would certainly draw a lot less power, anyway.)

[Mnky313]

49 minutes ago, LIGISTX said:

To elaborate on this, if it is in a VM, what hypervisor are you using? I know with proxmox I had to edit the CPU to not emulate, but to actually be “system” which allowed me to use hardware accelerated encryption/decryption within pfsense. 

AES-NI shows up an inactive if that has anything to do with it... (I believe it was the same on the old router as well)
 

 

Also how did you set the CPU to 'host' under processor settings on the VM? Or is there another step, I'm using proxmox on a separate system for different VMs & if I can improve performance that would be awesome.

[Mnky313]

Just now, Needfuldoer said:

What platform are you running?

 

Can you step it up to a pair of faster processors, like two E5-2667 v2s?

 

Depending on your firewall needs, you might be better off with a Haswell or Skylake era desktop with a couple extra NICs thrown into it. (That would certainly draw a lot less power, anyway.)

an old EMC server with an Intel Server Board S2600GZ, it should be compatible with v2 CPUs.

I am shifting servers around so I could swap some CPUs around (I have another system with 2667 v2s that could use some more cores because it's got a bunch of VMs on it).

I'm curious how they managed such great performance out of an atom though...

[Mnky313]

Interesting, I enabled PowerD and set it to maximum performance and speeds went up a bit (to ~250mbps)

Before I pick up faster single threaded CPUs I'll check the BIOS and see if there's some turbo boost/performance settings that are off, I can't imagine a ~8% (~1200 pts vs ~1110 pts) performance loss would cause a speeds to be halved...

[LIGISTX]

1 hour ago, Mnky313 said:

AES-NI shows up an inactive if that has anything to do with it... (I believe it was the same on the old router as well)
 

 

Also how did you set the CPU to 'host' under processor settings on the VM? Or is there another step, I'm using proxmox on a separate system for different VMs & if I can improve performance that would be awesome.

You enable it in proxmox under the VM’s hardware settings. You can pick CPU, and you select “host”. 
 

I am confused tho, is pfsense running bare metal or virtually? Based on that screenshot it looks to be bare metal, and it reports it as supporting AES-NI but it isn’t actually turned on in pfsense (you have to tell pfsense to use hardware encryption in….. general settings? Advanced settings? Google it, I forget). But, if it is running bare metal, the real question is why are you running pfsense on that machine? Are you supporting a multi thousand user deployment, lol. I run pfsense on 2 threads of my homelab, and it has way more headroom then needed. 

[Mnky313]

40 minutes ago, LIGISTX said:

I am confused tho, is pfsense running bare metal or virtually? Based on that screenshot it looks to be bare metal

pfsense is bare metal.

the proxmox system is a totally separate discussion.

 

Quote

 I'm using proxmox on a separate system for different VMs & if I can improve performance that would be awesome.

 

52 minutes ago, LIGISTX said:

and it reports it as supporting AES-NI but it isn’t actually turned on in pfsense (you have to tell pfsense to use hardware encryption in….. general settings? Advanced settings? Google it, I forget).

Yeah, I found it when I enabled PowerD, it's under advanced.

 

42 minutes ago, LIGISTX said:

the real question is why are you running pfsense on that machine? Are you supporting a multi thousand user deployment, lol. I run pfsense on 2 threads of my homelab, and it has way more headroom then needed. 

quite frankly, it was cheap and has interfaces to do what I need.

I prefer to use bare metal for the router rather than virtualizing it and this allowed me to put in a 10gbe sfp+ card I already had as well as multiple rj45 interfaces for other stuff like giving a set of VMs their own network that can't access other local PCs or keeping my stuff routed through a VPN while leaving the rest of the network unaffected.

 

[Mnky313]

4 hours ago, Mnky313 said:

So I recently moved pfSense to a new system (2x xeon e5-2620, 16gb RAM) (fresh install, not backup/restore) and wireguard speeds are significantly slower than both running the wireguard app on windows and connecting to the same server & the old system that had pfSense on it (fx-7600p, 4gb RAM).

I'm connecting to the same server on all tests:

without a vpn:

924mbps down
511mbps up

running wireguard on my windows pc/(speeds are similar on pfSense on the old hardware), this also flucuates between ~400-700mbps down most of the time:
564mbps down

133mbps up

 

running wireguard on pfSense:

158mbps down

196mbps up

 

I've also tried different servers and ports and that doesn't seem to make any difference.
(& MTU/MSS is set to 1420, tried lowering it to 1280 and that also didn't make a difference.)

Disabled Hyperthreading & switched power profile from 'Balance Power' to 'Performance' resolved the issue.

getting similar speeds now ~500Mbps, I'll keep testing to see If I can get even faster but that's good enough for me.

this also improved performance across that bridge I mentioned, went from ~2.7Gbps to ~5.8Gbps. Still not near 10Gbps but It's much better, I'll probably wait until I switch game servers over to a different box (because they're currently on the system with the 2667v2s) and steal the CPUs from it.

[LIGISTX]

20 minutes ago, Mnky313 said:

I prefer to use bare metal for the router rather than virtualizing it and this allowed me to put in a 10gbe sfp+ card I already had as well as multiple rj45 interfaces for other stuff like giving a set of VMs their own network that can't access other local PCs or keeping my stuff routed through a VPN while leaving the rest of the network unaffected.

You can save a lot of electricity by virtualizing. 
 

My pfsense is virtual and I have a many vlans and subnets. Quad 1gbe NIC and a Sfp+ card passed through to pfsense. 
 

I would at least pull one of the CPU’s out of the machine just to save on power if your adamant on staying bare metal. 

[Mnky313]

1 minute ago, LIGISTX said:

You can save a lot of electricity by virtualizing. 
 

My pfsense is virtual and I have a many vlans and subnets. Quad 1gbe NIC and a Sfp+ card passed through to pfsense. 
 

I would at least pull one of the CPU’s out of the machine just to save on power if your adamant on staying bare metal. 

I don't care much about power tbh.
I'd rather have separate machines to do critical stuff so when I need to mess with the main VM server it doesn't take out everything.

As for pulling a CPU out, I probably will only put one of the 2667 v2s in once I'm ready to move them to this server.

[LIGISTX]

4 minutes ago, Mnky313 said:

I don't care much about power tbh.
I'd rather have separate machines to do critical stuff so when I need to mess with the main VM server it doesn't take out everything.

As for pulling a CPU out, I probably will only put one of the 2667 v2s in once I'm ready to move them to this server.

That’s fair. I used to run pfsense bare metal for this reason. But I figured no point is having 2 machines on 24/7 when my homelab has plenty of power to support a pfsense VM. And I do have a test bench I can always throw pfsense on in case I needed to down the homelab for maintenance/changes. 

[Alex Atkin UK]

8 hours ago, LIGISTX said:

That’s fair. I used to run pfsense bare metal for this reason. But I figured no point is having 2 machines on 24/7 when my homelab has plenty of power to support a pfsense VM. And I do have a test bench I can always throw pfsense on in case I needed to down the homelab for maintenance/changes. 

Yeah same here, its why I use appliances based on laptop components, it keeps the power of bare-metal boxes so low you wouldn't really be saving much, going VM.  I'm actually hoping to run them off PoE at some point as that tends to save a lot compared to the crap PSUs they come with too, while saving an outlet, but as they can peak above 3A the few PoE splitters on the market that support over 36W are stupidly expensive.

Also, the boxes I have going 24/7 I sometimes need all the CPU power to be available, which I wouldn't want to be stealing from my router VM.

 

In addition, I believe Wireguard only uses software encryption/decrytion due to it using supposedly more secure encryptions schemes rather than AES.  So unless its multi-threaded, its going to be bottlenecked.  Still in some cases its faster than OpenVPN (which has other CPU bottlenecks other than encryption) despite, but I've also seen people report it slower, so not sure what specific CPUs its good/bad on.