10gb file transfer help!

[K1ngofportal]

Hello to my favorite tech family,

I seem to be having a problem with my server and file transfer speeds. So I just got two Mellanox connects-3 10GbE cards to run from my PC to my server and I seem to be hitting a bottle neck. I’m only getting 500MB/s file transfer I’ve tried turning off my firewall and no luck…. Any help would be great! 
 

Here are the specs of both PCs 

 

Server,

OS Unraid

MB intel s5520hc

Cpu Xeon x5650 x2

ram 16gb

x2 512 sata ssd in raid 0 as cache 

x10 WD 4tb red/gold 

HBA lsi sas9211-8i 

 

the two ssd are plugged into the LSI card

 

 

PC,

OS windows 10

MB asus B550-f gaming WiFi

CPU Ryzen 7 5800x

Ram 32gb

Nvme 4.0 Corsair p600 500

RTX 3080

 

if there anything else let me know, any and all idea are welcome

[Eigenvektor]

I would assume you're being bottlenecked by the read and/or write speed of the disks. Unless data is sequential, you'll not get max read speed. And if the data to transfer is not on the SSD cache, you'll be limited by the HDDs speed. If the data is in the cache, you're likely limited by the RAID controller's maximum speed.

[FloRolf]

Well first off, windows is dumb and for some reason can't handle 10g properly. 

BUT my experience is that P2P should actually get pretty close. Can you run a quick iperf test to verify you actually have a full 10g link? 

Secondly, what's up with that Raid0 cache? That seems a little weird tbh. 1tb of cache that will probably barely get any use? Personally not a big fan of cache like that. Can you try running the Raid0 as an indipendent array and run a speed test? 

 

 

Edit: here's an example of another recent post. The dude also only gets about 500mb/s write speed on windows but much more on mac or Linux. 

 

[Skiiwee29]

@K1ngofportal I have merged your threads together. Please keep to a single thread for the same problem. 

[K1ngofportal]

45 minutes ago, Eigenvektor said:

I would assume you're being bottlenecked by the read and/or write speed of the disks. Unless data is sequential, you'll not get max read speed. And if the data to transfer is not on the SSD cache, you'll be limited by the HDDs speed. If the data is in the cache, you're likely limited by the RAID controller's maximum speed.

I’m writing straight to the ssd that are in raid 0 and the raid controller is just unraid software.

[K1ngofportal]

24 minutes ago, Skiiwee29 said:

@K1ngofportal I have merged your threads together. Please keep to a single thread for the same problem. 

Oh thank you and sorry didn’t know that.

[K1ngofportal]

33 minutes ago, FloRolf said:

Well first off, windows is dumb and for some reason can't handle 10g properly. 

BUT my experience is that P2P should actually get pretty close. Can you run a quick iperf test to verify you actually have a full 10g link? 

Secondly, what's up with that Raid0 cache? That seems a little weird tbh. 1tb of cache that will probably barely get any use? Personally not a big fan of cache like that. Can you try running the Raid0 as an indipendent array and run a speed test? 

 

 

Edit: here's an example of another recent post. The dude also only gets about 500mb/s write speed on windows but much more on mac or Linux. 

 

Windows is dumb with a lot of things lol but I’ll run the iperf test when I get home and see if I’m able to run at 10g speeds, as for the raid 0 cache it was just for testing purposes I actually plan on getting an 128g m.2 for the cache I just had the ssd from some computers I’m trying to sell. 

[Eigenvektor]

1 hour ago, K1ngofportal said:

I’m writing straight to the ssd that are in raid 0 and the raid controller is just unraid software.

So you're saying the HBA lsi sas9211-8i raid controller in your hardware specs isn't being used?

 

Have you run a speed test on the SSDs? Write speed of a Raid 0 isn't necessarily double that of the individual disks.

[leadeater]

3 hours ago, FloRolf said:

Edit: here's an example of another recent post. The dude also only gets about 500mb/s write speed on windows but much more on mac or Linux. 

This is almost always due to Linux SAMBA implementation and optimization, it's not a fault with Windows. Windows to Windows will do many GB/s.

 

You can fix the above with either enabling SMB multichannel or SMB Direct, both of which automatically detect and work when going between Windows to Windows. SMB is first and foremost Microsoft's thing, Linux has always had terrible SMB support and performance because they are for the most part having to reverse engineer everything. Apple in their so infinite wisdom developed their own SMB implementation rather than using SAMBA and contributing to that project.

 

Quote

This paper provided performance results for SMB 3.1.1 running over Chelsio’s T6 iWARP RDMA enabled Ethernet adapter, T62100-CR. Chelsio iWARP RDMA solution delivers ground breaking 95Gbps line rate performance and 530K IOPs using standard Ethernet infrastructure. With plugand-play operation, enhanced reliability, higher efficiency and line rate performance, SMB over Chelsio’s T6 series of RDMA enabled adapters is an extremely compelling solution for Windows Server 2016 storage networking.

https://www.chelsio.com/wp-content/uploads/resources/T5-100G-SMB-Windows.pdf

Windows will happily do near line rate 100Gbps SMB for example.

 

Sure not so helpful information for this topic or the other one, other than get SMB Direct working between Linux and Windows and you can achieve the above.

 

2 hours ago, K1ngofportal said:

Windows is dumb with a lot of things lol but I’ll run the iperf test when I get home and see if I’m able to run at 10g speeds, as for the raid 0 cache it was just for testing purposes I actually plan on getting an 128g m.2 for the cache I just had the ssd from some computers I’m trying to sell. 

Use a tool like IOmeter to do your performance tests, it's much easier to control and get consistent tests this way and you can test different I/O sizes to figure out where any performance issues are