Expanding Truenas Server

[Hette]

I have an existing an running configuration from an older lenovo c30:

dual CPU Xeon e5-2860

128 gb of ecc ram ddr3

1 GB Ethernet

bootpool 2 mirrored ssd drives over SATA onboard

pool1 2 mirrored 8 TB SATA drives WD red

pool2 1 8 TB SATA drive WD red gets resynced with pool 1

the HDDs are connected over a 5 port SATA 3.0 x4 pci3.0 adapter

 

i get very inconsistent data out- and input. Sometimes it‘s just around 100 MB/s but suddenly drops to a few KB/s.

i was wondering if i could improve de datarate significantly if i would create a pool 3 with ssd

 

i was thinking something like this: https://www.asus.com/ch-en/Motherboards-Components/Motherboards/Accessories/HYPER-M-2-X16-CARD-V2/

or: https://www.digitec.ch/en/s1/product/microconnect-pci-e-88se9215-8-port-sata-controller-cards-18153677?gclid=EAIaIQobChMIoYixvpSc-QIVFeuyCh1_1wB9EAQYBiABEgKDQvD_BwE&gclsrc=aw.ds

 

i‘m just not sure if the x4 lines are fast enough. Should i go with x16 or is ist simply and overkill for the system? The CPUs and RAM are never maxed out.

 

 I would also need a new ethernet adaptor like: https://www.amazon.com/ASUS-XG-C100C-Network-Adapter-Single/dp/B072N84DG6/ref=sr_1_2?keywords=10gb+network+card&qid=1659031288&sr=8-2

 

i‘m a bit overwhelmed by the possible choices and i‘m not sure if these choices would really improve the system that much overall. What is your opinion?

[AbydosOne]

2 hours ago, Hette said:

i get very inconsistent data out- and input. Sometimes it‘s just around 100 MB/s but suddenly drops to a few KB/s.

i was wondering if i could improve de datarate significantly if i would create a pool 3 with ssd

Your issue is probably related to the SATA card. TrueNAS doesn't play well (in my experience) with consumer grade SATA cards (specifically ones with Marvell controllers). Adding SSDs for sure won't help.

 

You're probably better off getting an older LSI HBA, which have much better compatibility with TrueNAS.

 

A 10G NIC also won't improve speeds much for a single mirrored array.

[Hette]

2 hours ago, AbydosOne said:

Your issue is probably related to the SATA card. TrueNAS doesn't play well (in my experience) with consumer grade SATA cards (specifically ones with Marvell controllers). Adding SSDs for sure won't help.

 

You're probably better off getting an older LSI HBA, which have much better compatibility with TrueNAS.

 

A 10G NIC also won't improve speeds much for a single mirrored array.

my idea was to leave the mirrored pool for long term storage. And build a new pool consisting of 4 to 5 SSD.

 

it‘s really confusing to pick the right HBA. I‘m not really understanding what things i have to look out for. It must not be a raid controller as far as i understand but then my knowledge ends…

What do you think about such a thing: Supermicro LSI SAS 3008 IT/HBA MODE 12GB

[AbydosOne]

8 minutes ago, Hette said:

What do you think about such a thing: Supermicro LSI SAS 3008 IT/HBA MODE 12GB

For SATA drives, I don't see a reason to go quite that high-end. I use a used LSI 9207-8i, which you can get flashed to IT mode with breakout cables for ~$50 (look around eBay for the best deal, that's just a quick example).

 

12 minutes ago, Hette said:

my idea was to leave the mirrored pool for long term storage. And build a new pool consisting of 4 to 5 SSD.

Unless you really need multiple gigabytes per second of throughput, it's not what's going to solve your issues. Furthermore, I doubt that a) the Lenovo C30 has PCIe gen 3, let alone gen 4 support, and b) it supports bifurcation to use the Hyper card. Just avoid it unless you know you truly need it.

[cmndr]

As hinted at, there's a good chance that your performance is getting mucked around by the HBA.

I have a set up (slow CPU, 32GB RAM, 118GB optane cache, 4HDDs in RAIDz1) where reads are usually 10-700MBps (limited by a pcie x1 slot) as are writes.

 

With as much RAM as you have, I don't think it's the hardware (at least not for reads).

 

 

 

Are you doing Sync writes? If you are THAT could affect performance and it'd be a decent use case for a SLOG device (16GB optane stick could handle that for a 1Gbe NIC).  In general you should NOT be getting KBps on a ZFS based system. Single digit MBps maybe.

 

 

---------

 

 

As a general suggestion - if you're trying to go 10Gbe, SFP+ is a lot cheaper if you don't need to go a long distance.
https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m570.l1313&_nkw=connectx+3&_sacat=0

This lets you use switches like the Mikrotik crs305-1g-4s+in or QNAP QSW-M408-2C, or QNAP QSW-308-1C (not managed, you might not need management)

 

You can still do RJ45 10Gbe for longer runs or on "normal" clients like a desktop.

 

[Hette]

Currently i’m using this SATA adapter: https://www.delock.com/produkt/90395/merkmale.html

Quote

I doubt that a) the Lenovo C30 has PCIe gen 3, let alone gen 4 support, and b) it supports bifurcation to use the Hyper card.

The C30 does have PCI 3.0 support.

 

Quote

 

Are you doing Sync writes? If you are THAT could affect performance and it'd be a decent use case for a SLOG device (16GB optane stick could handle that for a 1Gbe NIC).  In general you should NOT be getting KBps on a ZFS based system. Single digit MBps maybe.

What is sync writes? I‘m not entirely sure i‘m not doing this. Where is the setting for that?

i thought with 128 GB of RAM this shouldn‘t really be the problem as the buffer for writes should be big enough.

[Kilrah]

15 minutes ago, Hette said:

i thought with 128 GB of RAM this shouldn‘t really be the problem as the buffer for writes should be big enough.

RAM is mostly used for read caching, write cache is very limited for obvious reasons of data being at the mercy of any kind of failure for the whole time it would take to flush the cache to the disks.

 

Sync writes are almost like disabling write cache, a transfer will not be considered completed until fully written to disk. Can slow things down a lot.

https://www.45drives.com/community/articles/zfs-caching/

[LIGISTX]

Agreed to the “get a real HBA” comments. LSI flashed to IT mode, eBay is full of them. My Dell H310 has worked great for myself and others. 
 

Also, no need for SSD’s especially over gigabit. A decently sized harddrive pool can easily saturate gigabit.