Zpool optimization

[wingyiulam]

Hi all,

 

I am new to ZFS. My zpool doesn't saturate a 10gbe port, but ipref3 shows that the 10gbe has been saturated. Can you guys give me some advices?

 

A single drive performance of ST16000NM001G in my server is about 100up & down with raidz2, and it's poorly performed. the sector on each drive has been converted to 4Kn. (The Data sheet stated that "SEQ R & W 261/249" IOPS "170/440" ) 

 

I also don't see any performance gain running an Optane SSD......all my equipments are connected to a UPS; I don't know if I need further protection. should I simply lose this expensive SSD and get a cheap 800p instead?

 

Sequential read and write are only my concerned for my daily workflow.

 

spec:

Giagbyte x570s Aorus Master

Kingston 32GB DDR4-3200 ECC UDIMM KSM32ED8/32ME

AMD Ryzen 3700X

LSI 9201-16i

Intel X540-T2

10 x ST16000NM001G (4KN) running Raidz2

1 x INTEL SSDPEL1K100GA for SLOG

No L2ARC (not planning to use it until i need it)

 

Zpool Spec:

ashift=12

compression=lz4

recordsize=1M

xattr=sa

dnodesize=auto

atime=off

sync=standard

 

Jumbo frame is at 9000MTU

ipref: 10gbe per sec

Mac: 800mb write/ 750mb read

Win: 450mb write/ 800mb read (the write speed in Windows is even worst)

 

Thanks,

Wing

 

[Electronics Wizardy]

2 minutes ago, wingyiulam said:

Jumbo frame is at 9000MTU

ipref: 10gbe per sec

Mac: 800mb write/ 750mb read

Win: 450mb write/ 800mb read (the write speed in Windows is even worst)

 

This feels like smb tuning here. Try running fio on the server to see what performance it gets?

 

What is disk usage on the system when running tests?

 

3 minutes ago, wingyiulam said:

1 x INTEL SSDPEL1K100GA for SLOG

Are you doing sync writes? Id just turn sync off and remove this for most uses.

 

 

[wingyiulam]

3 minutes ago, Electronics Wizardy said:

This feels like smb tuning here. Try running fio on the server to see what performance it gets?

 

What is disk usage on the system when running tests?

 

Are you doing sync writes? Id just turn sync off and remove this for most uses.

 

 

i see, I will give it a try. its a new build.

i don't know if I should do sync write; the write performance is suffer whenever I set "sync to always".

[Electronics Wizardy]

Just now, wingyiulam said:

i see, I will give it a try. its a new build.

i don't know if I should do sync write; the write performance is suffer whenever I set "sync to always".

Normally SMB doesn't use sync. Id leave sync off. It will give the best performance, and normally isn't needed here. Worst case you lose the last 5 seconds of data before a crash

[leadeater]

Additionally for 10 HDDs what you are getting is near as much the best those could do so beyond SMB tuning like getting SMB Direct working I don't think you're going to gain too much performance.

[FloRolf]

I've honestly tinkered with Windows and 10gbe so much that I can say windows is just a b*tch. It tried everything I could find online and it still had some hard stop bottleneck somewhere (more drastical than yours though!) 

What finally solved it for me is really janky but it works so I am not going to change it lol. 

I have a 540-T2 in both PC and NAS, one cable goes through a 10g switch and the other goes P2P between the two machines. That P2P was never set up and windows just decided to use it for load balancing so the connection between the two is always perfectly balanced between the two NICs. Not sure why/how but I get 10g. Maybe this helps lol. 

[leadeater]

23 hours ago, FloRolf said:

I have a 540-T2 in both PC and NAS, one cable goes through a 10g switch and the other goes P2P between the two machines. That P2P was never set up and windows just decided to use it for load balancing so the connection between the two is always perfectly balanced between the two NICs. Not sure why/how but I get 10g. Maybe this helps lol. 

SMB Multichannel 

 

Run the PowerShell command (as Admin)

Get-SmbMultichannelConnection

 

[jde3]

Raidz2 (raidz in general) isn't really used for performance, it's used for capacity.

 

You can get really good performance out of ZFS but you'll want to use mirrored arrays and a cache drive. The ARC is your friend and if you can keep your workload in the ARC you'll be doing great.

Something like this:
 

config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	    da11    ONLINE       0     0     0
	  mirror-2  ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da12    ONLINE       0     0     0
	  mirror-3  ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da13    ONLINE       0     0     0
	  mirror-4  ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da14    ONLINE       0     0     0
	spares
	    da15      AVAIL
    logs
        nvme0n1p2	ONLINE       0     0     0
    cache
        nvme0n1p1	ONLINE       0     0     0