TrueNas Server build only writing at 115Mb/s

[Brett M]

Hello all,

 

I built 2 server's copying the Mark Rober build from youtube with linus tech. When I try to write files to it, it caps out at 115Mb/s. I know this is slower than it should be, because I have a synology NAS with worse specs that will do 350Mb/s. My components are almost exactly the ones used in the video. Motherboard: ROMED8-2T, Epyc 16core CPU, 128gb DDR4 Ram with 12, 20Tb HDD at 7200rpm 6Gb/s (Seagate Exos X22 SATA). I have never used TrueNas before am an wondering if there is a setting I need to change to read and write faster? It should not be transferring this slow because our motherboard has dual 10Gb Ethernet ports. Please Help!

[Needfuldoer]

4 minutes ago, Brett M said:

It should not be transferring this slow because our motherboard has dual 10Gb Ethernet ports

They're connected to other 10Gb interfaces and negotiated a 10Gb link? There's no plain ol' Gigabit link anywhere between the client and the server?

 

115 MB/sec is about what I'd expect to see on an otherwise idle Gigabit link.

[Robchil]

not realy surprised..  hdd's are not fast.. no use for 10Gb nic's if your storage is hdd's... regarless of interface.  what RAID are you running? 

 

depends on what RAID you use but synology can spread data on several drives with it's own RAID.. to get closer to striping speed... but still.. hdd's are not fast.. 115Mb/s is faster than data on single drive transfer.. 

 

 

[Brett M]

Hi, yes I connected them directly to each other, set static ip addresses on the computer and NAS. We're running RaidZ2 with 6 hdds. These are the same speed hard drives that we have in our Synology NAS.  By doing the same thing with our existing Synology NAS we're getting 300-315 MB/s. We also have a 1 TB NVMe cache drive in our new NAS.

[Brett M]

On 4/30/2024 at 4:55 PM, Robchil said:

not realy surprised..  hdd's are not fast.. no use for 10Gb nic's if your storage is hdd's... regarless of interface.  what RAID are you running? 

 

depends on what RAID you use but synology can spread data on several drives with it's own RAID.. to get closer to striping speed... but still.. hdd's are not fast.. 115Mb/s is faster than data on single drive transfer.. 

 

 

Hi, yes I connected them directly to each other, set static ip addresses on the computer and NAS. We're running RaidZ2 with 6 hdds. These are the same speed hard drives that we have in our Synology NAS.  By doing the same thing with our existing Synology NAS we're getting 300-315 MB/s. We also have a 1 TB NVMe cache drive in our new NAS.

[Brett M]

On 4/30/2024 at 1:55 PM, Needfuldoer said:

They're connected to other 10Gb interfaces and negotiated a 10Gb link? There's no plain ol' Gigabit link anywhere between the client and the server?

 

115 MB/sec is about what I'd expect to see on an otherwise idle Gigabit link.

Hi, yes I connected them directly to each other, set static ip addresses on the computer and NAS. We're running RaidZ2 with 6 hdds. These are the same speed hard drives that we have in our Synology NAS.  By doing the same thing with our existing Synology NAS we're getting 300-315 MB/s. We also have a 1 TB NVMe cache drive in our new NAS.

[Robchil]

6 hours ago, Brett M said:

Hi, yes I connected them directly to each other, set static ip addresses on the computer and NAS. We're running RaidZ2 with 6 hdds. These are the same speed hard drives that we have in our Synology NAS.  By doing the same thing with our existing Synology NAS we're getting 300-315 MB/s. We also have a 1 TB NVMe cache drive in our new NAS.

so raid 6.. basicaly raid 5 with 2 failover disks..  unless it's raid 0... you won't realy get magicaly faster speeds from a hdd.. sas or sata.. with raid0 you can get up to 2 times. 

there are a few limiters you have to consider..

1.. the disks max out between 80-150MB/s if it's large datafiles.. 

2.. what bandwith are your raid controller able to push?. a normal dell perc6 card are limited to ca 300MB/s or 2400Mb/s .. bits.. not Bytes. 

3. what filesizes are you working with? alot of small files will slow down a transfer.. regardless.. often down to 10-20MB/s

4. what controller are your nvme on? sata or PCIE?.. if it's on the sata controller your stuck with max 550MB/s on it.. PCIE what version? 3 can give up to 3000MB/s. 

 

there are alot of links... that can slow down the chain.. just like internet.. data transfer are never faster than the slowest component. 

[Brett M]

20 hours ago, Robchil said:

so raid 6.. basicaly raid 5 with 2 failover disks..  unless it's raid 0... you won't realy get magicaly faster speeds from a hdd.. sas or sata.. with raid0 you can get up to 2 times. 

there are a few limiters you have to consider..

1.. the disks max out between 80-150MB/s if it's large datafiles.. 

2.. what bandwith are your raid controller able to push?. a normal dell perc6 card are limited to ca 300MB/s or 2400Mb/s .. bits.. not Bytes. 

3. what filesizes are you working with? alot of small files will slow down a transfer.. regardless.. often down to 10-20MB/s

4. what controller are your nvme on? sata or PCIE?.. if it's on the sata controller your stuck with max 550MB/s on it.. PCIE what version? 3 can give up to 3000MB/s. 

 

there are alot of links... that can slow down the chain.. just like internet.. data transfer are never faster than the slowest component. 

1. Ok, sounds good on that one.

2. 8 of our hard drives are connected straight to the motherboard using 2 SAS to SATA cables/splitters and the other 4 are connected using the same cable onto an LSI SAS 9300-16I 12GB/S SATA+SAS HBA Host Bus Adapter Card.

3. We're working with fairly large file sizes, mostly RAW video files to be edited using Premier Pro.

4. Both of the nvme drives are connected directly to the two motherboard slots.

I did just figure out that my 10GB switch was auto negotiating the link speeds to 1 Gbps for some reason,  so I connected directly between my tower and the NAS (which I thought I had already tried that) and our transfer speeds improved to around 315 MB/s, which is what our Synology was doing, but we were hoping for a little bit faster speeds. Thanks a lot for the help!

[Robchil]

12 hours ago, Brett M said:

1. Ok, sounds good on that one.

2. 8 of our hard drives are connected straight to the motherboard using 2 SAS to SATA cables/splitters and the other 4 are connected using the same cable onto an LSI SAS 9300-16I 12GB/S SATA+SAS HBA Host Bus Adapter Card.

3. We're working with fairly large file sizes, mostly RAW video files to be edited using Premier Pro.

4. Both of the nvme drives are connected directly to the two motherboard slots.

I did just figure out that my 10GB switch was auto negotiating the link speeds to 1 Gbps for some reason,  so I connected directly between my tower and the NAS (which I thought I had already tried that) and our transfer speeds improved to around 315 MB/s, which is what our Synology was doing, but we were hoping for a little bit faster speeds. Thanks a lot for the help!

well.. the hardware are the limiting factors here.. if you go all nvme raid you'll see different speeds

and i would recommend a fiber 10GB fiber nic and a twinax cable