Jump to content

25GbE - Strange Windows Issue - Downloads Only

Lurick
By Lurick
in Networking
 Share
Go to solution Solved by leadeater,
11 hours ago, Lurick said:

Rebooted into Safe Mode w/ Networking and voila 24.5Gbit each way so something in Windows is definitely mucking things up.

Well that's odd, that it's slower in normal.

 

Turned off Defender Real Time Protection? Defender does do network inspection which you can disable individually with GPO etc

 

In safe mode defender doesn't work so that is your most likely cause I can think of right now.

 

Past that make sure you have looked at the NIC hardware properties in the Windows Drivers and made sure all the offloads are actually enabled and push up the buffer sizes etc, maybe the driver defaults are different in safe mode but I highly doubt that. I do find increasing buffers/ring sizes helps a lot though.

Posted

I'm a bit perplexed at this point but figured it's worth posting here just in case someone might have an idea of what I'm missing since this seems to be a Windows issue but I'm not 100% sure.

 

Summary:

I've got two 25GbE NICs connected to a 25GbE capable switch. Links negotiate at 25GbE and uploads from Windows to the remote servers pass at 25GbE with iperf2 and iperf3 (about 24Gb/s with iperf2 and 21 with iperf3) but downloads are the issue from the remote server to my Windows machine. No matter what I've tried I can't seem to break some mystical sub-13Gb/s barrier and I'm kind of lost as to why.

 

What I've tried:

I've tried disabling all Windows security features (Firewall, anti-virus, etc) and no real change, I'm capped at ~12.5Gb/s downloads.

Reinstalling NIC drivers and uninstalling drivers/rebooting/reinstalling but no change, still about 12.5Gb/s downloads.

Testing different networks and on the same network as the VM and no change.

Even two VMs on different VLANs that need to cross the switch work in both directions without issue and hit 24.5Gb/s in both directions without issue so something seems to be up with my machine only.

 

I see no output drops on the switch and everything looks normal so that's what has me suspecting a Windows issue. I've not tried safe mode with networking just yet but that's next on my list. Was just hoping anyone like @mynameisjuan or @leadeater or anyone else might have any ideas as to why downloads would be the issue on Windows.

 

Specs:

NIC: Intel XXV710

Switch: Catalyst 9300X-48HX with 8x25GbE module.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

How remote are the servers? Are they on a connection with an upload limit?

If you're interested in a product please download and read the manual first.

Don't forget to tag or quote in your reply if you want me to know you've answered or have another question.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
14 minutes ago, keskparane said:

How remote are the servers? Are they on a connection with an upload limit?

They're local, to the same switch

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
15 minutes ago, Lurick said:

They're local, to the same switch

so odd question is data at some point going thru your isp router?

in the transfer over the network

MSI x399 sli plus  | AMD theardripper 2990wx all core 3ghz lock |Thermaltake flo ring 360 | EVGA 2080, Zotac 2080 |Gskill Ripjaws 128GB 3000 MHz | Corsair RM1200i |150tb | Asus tuff gaming mid tower| 10gb NIC

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

What model of NICs are you using?

Main System (Byarlant): Ryzen 7 5800X | Asus B550-Creator ProArt | EK 240mm Basic AIO | 16GB G.Skill DDR4 3200MT/s CAS-14 | XFX Speedster SWFT 210 RX 6600 | Samsung 990 PRO 2TB / Samsung 960 PRO 512GB / 4× Crucial MX500 2TB (RAID-0) | Corsair RM750X | Mellanox ConnectX-3 10G NIC | Inateck USB 3.0 Card | Hyte Y60 Case | Dell U3415W Monitor | Keychron K4 Brown (white backlight)

 

Laptop (Narrative): Lenovo Flex 5 81X20005US | Ryzen 5 4500U | 16GB RAM (soldered) | Vega 6 Graphics | SKHynix P31 1TB NVMe SSD | Intel AX200 Wifi (all-around awesome machine)

 

Proxmox Server (Veda): Ryzen 7 3800XT | AsRock Rack X470D4U | Corsair H80i v2 | 64GB Micron DDR4 ECC 3200MT/s | 4x 10TB WD Whites / 4x 14TB Seagate Exos / 2× Samsung PM963a 960GB SSD | Seasonic Prime Fanless 500W | Intel X540-T2 10G NIC | LSI 9207-8i HBA | Fractal Design Node 804 Case (side panels swapped to show off drives) | VMs: TrueNAS Scale; Ubuntu Server (PiHole/PiVPN/NGINX?); Windows 10 Pro; Ubuntu Server (Apache/MySQL)


Media Center/Video Capture (Jesta Cannon): Ryzen 5 1600X | ASRock B450M Pro4 R2.0 | Noctua NH-L12S | 16GB Crucial DDR4 3200MT/s CAS-22 | EVGA GTX750Ti SC | UMIS NVMe SSD 256GB / Seagate 1.5TB HDD | Corsair CX450M | Viewcast Osprey 260e Video Capture | Mellanox ConnectX-2 10G NIC | LG UH12NS30 BD-ROM | Silverstone Sugo SG-11 Case | Sony XR65A80K

 

Camera: Sony ɑ7II w/ Meike Grip | Sony SEL24240 | Samyang 35mm ƒ/2.8 | Sony SEL50F18F | Sony SEL2870 (kit lens) | PNY Elite Perfomance 512GB SDXC card

 

Network:

Spoiler 
                           ┌─────────────── Office/Rack ────────────────────────────────────────────────────────────────────────────┐
Google Fiber Webpass ────── UniFi Security Gateway ─── UniFi Switch 8-60W ─┬─ UniFi Switch Flex XG ═╦═ Veda (Proxmox Virtual Switch)
(500Mbps↑/500Mbps↓)                             UniFi CloudKey Gen2 (PoE) ─┴─ Veda (IPMI)           ╠═ Veda-NAS (HW Passthrough NIC)
╔═══════════════════════════════════════════════════════════════════════════════════════════════════╩═ Narrative (Asus USB 2.5G NIC)
║ ┌────── Closet ──────┐   ┌─────────────── Bedroom ──────────────────────────────────────────────────────┐
╚═ UniFi Switch Flex XG ═╤═ UniFi Switch Flex XG ═╦═ Byarlant
   (PoE)                 │                        ╠═ Narrative (Cable Matters USB-PD 2.5G Ethernet Dongle)
                         │                        ╚═ Jesta Cannon*
                         │ ┌─────────────── Media Center ──────────────────────────────────┐
Notes:                   └─ UniFi Switch 8 ─────────┬─ UniFi Access Point nanoHD (PoE)
═══ is Multi-Gigabit                                ├─ Sony Playstation 4 
─── is Gigabit                                      ├─ Pioneer VSX-S520
* = cable passed to Bedroom from Media Center       ├─ Sony XR65A80K (Google TV)
** = cable passed from Media Center to Bedroom      └─ Work Laptop** (Startech USB-PD Dock)

 

Retired/Other:

Spoiler

Laptop (Rozen-Zulu): Sony VAIO VPCF13WFX | Core i7-740QM | 8GB Patriot DDR3 | GT 425M | Samsung 850EVO 250GB SSD | Blu-ray Drive | Intel 7260 Wifi (lived a good life, retired with honor)

Testbed/Old Desktop (Kshatriya): Xeon X5470 @ 4.0GHz | ZALMAN CNPS9500 | Gigabyte EP45-UD3L | 8GB Nanya DDR2 400MHz | XFX HD6870 DD | OCZ Vertex 3 Max-IOPS 120GB | Corsair CX430M | HooToo USB 3.0 PCIe Card | Osprey 230 Video Capture | NZXT H230 Case

TrueNAS Server (La Vie en Rose): Xeon E3-1241v3 | Supermicro X10SLL-F | Corsair H60 | 32GB Micron DDR3L ECC 1600MHz | 1x Kingston 16GB SSD / Crucial MX500 500GB

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
5 minutes ago, AbydosOne said:

What model of NICs are you using?

Intel xxv710 NICs on both sides.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Rebooted into Safe Mode w/ Networking and voila 24.5Gbit each way so something in Windows is definitely mucking things up.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 hour ago, Lurick said:

Rebooted into Safe Mode w/ Networking and voila 24.5Gbit each way so something in Windows is definitely mucking things up.

As you know, anything on the host OS side is out of my expertise. That said, I know that Safe Mode w/ Networking does load the network drivers so I would have to assume this is a service/program causing this.

 

I know Windows is terrible when it comes down to tshooing services/processes, but is there anything that stands out when performing the test in Task Manager/Resource Monitor?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
28 minutes ago, mynameisjuan said:

As you know, anything on the host OS side is out of my expertise. That said, I know that Safe Mode w/ Networking does load the network drivers so I would have to assume this is a service/program causing this.

 

I know Windows is terrible when it comes down to tshooing services/processes, but is there anything that stands out when performing the test in Task Manager/Resource Monitor?

Yah, I'm not great at the OS side myself but I did check on resource monitor and nothing spikes usage wise sadly which has me really perplexed since I would expect something to show up there if it was the limiting factor =/

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Solution
11 hours ago, Lurick said:

Rebooted into Safe Mode w/ Networking and voila 24.5Gbit each way so something in Windows is definitely mucking things up.

Well that's odd, that it's slower in normal.

 

Turned off Defender Real Time Protection? Defender does do network inspection which you can disable individually with GPO etc

 

In safe mode defender doesn't work so that is your most likely cause I can think of right now.

 

Past that make sure you have looked at the NIC hardware properties in the Windows Drivers and made sure all the offloads are actually enabled and push up the buffer sizes etc, maybe the driver defaults are different in safe mode but I highly doubt that. I do find increasing buffers/ring sizes helps a lot though.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
2 minutes ago, leadeater said:

Post what your iperf commands you are running, I'll do the same ones on some of my systems and see what I get.

Sure thing, I did disable Windows Defender real time protection but couldn't find the GPO for network inspection, do you happen to know where that might live?

 

iperf3 -s

iperf3 -c server -P 4 -R

I tried -P 6 as well but no change.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

I also have my NICs in a Windows Team and multiple Team Interfaces on tagged VLANs

 

image.png.6ca17a5aa78db256db17654d1bd2e6f2.png

 

 

image.png.a13f874b9871944b1503f00b9bf021d5.png

 

image.png.2966cca9b60ec2e6d25675fd82f34e72.png

 

Flow Control: Disabled

IPV4 Checksum Offload: RX & TX

Jumbo Packet 9000

Large Send Offload V2: Enabled

Max RSS Processes & Queues: 8

NetworkDirect: Enabled & Default

Packet Direct: Enabled

Quality of Service: Disabled

Receive Side Scaling: Enabled

Recv Seg Coalescing: Enabled 

Receive Buffers: 512

Send Buffers: 2048

TCP/UDP Checksum Offload: Enabled

 

Mellanox btw

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

@leadeater I found the issue 😄

webthreatdefsvc and webthreatdefusersvr - disabled the one that was running in services and boom, 24.5Gbit no problem!

 

Edit:

Forgot to mention these run even with Defender disabled.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
31 minutes ago, Lurick said:

@leadeater I found the issue 😄

webthreatdefsvc and webthreatdefusersvr - disabled the one that was running in services and boom, 24.5Gbit no problem!

 

Edit:

Forgot to mention these run even with Defender disabled.

What version of Windows are you running and is it also onboarded with Defender for Endpoint and has MsSenses.exe running?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
12 hours ago, leadeater said:

What version of Windows are you running and is it also onboarded with Defender for Endpoint and has MsSenses.exe running?

Windows 11 Pro

No defender for endpoint.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Networking

×