How to improve nfs IO speed

[tu_114_undercarriage]

Hey guys. I symboled a new NFS server with 6 Intel P4600 compose a soft RAID 0 array by `mdadm` in my studio as a NFS server. And my workstation connect to NFS server by 40G network. Both OS of server and client are Debian 10, and my NIC is Mellanox MCX354A-FCB.

I use command

fio --filename=test_file --name=seqread --rw=read --direct=1 --ioengine=libaio --bs=128k --size=100G --runtime=120 --group_reporting --iodepth=64 

to test io speed on NFS server, the speed of transmisiion is about 12.1GB/s. And then I mount NFS to client server, and use the same command to test the io speed of NFS, the result is 1.5GB/s. But theoretically the io speed should over 4GB/s. And executed same command on Server, and the result is 18.2GB/s.

So I want to konw what limits the io speed of NFS, and how can I adjust it?

Thankyou in advance!

[asquirrel]

Are you using jumbo frames (MTU 9000)? If not, that will help a good bit.

[tu_114_undercarriage]

12 hours ago, asquirrel said:

Are you using jumbo frames (MTU 9000)? If not, that will help a good bit.

Nope I'll try it later thx.

[Electronics Wizardy]

what speed does iperf say?

 

What cpu is on both ends?

 

Do you have rdma setup? 

 

[tu_114_undercarriage]

1 hour ago, Electronics Wizardy said:

what speed does iperf say?

 

What cpu is on both ends?

 

Do you have rdma setup? 

 

Thanks for your reply! And this is my iperf result:

username@hostname:~$ iperf -c 192.168.99.96 -p 50000
------------------------------------------------------------
Client connecting to 192.168.99.96, TCP port 50000
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.99.102 port 47914 connected with 192.168.99.96 port 50000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  18.2 GBytes  15.7 Gbits/sec

NFS Server, the CPU is Xeon E5-2680 v2. For client the CPU is Xeon CPU E7-8891 v3.

RDMA is not be set.

[Electronics Wizardy]

8 hours ago, tu_114_undercarriage said:

Thanks for your reply! And this is my iperf result:

username@hostname:~$ iperf -c 192.168.99.96 -p 50000
------------------------------------------------------------
Client connecting to 192.168.99.96, TCP port 50000
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.99.102 port 47914 connected with 192.168.99.96 port 50000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  18.2 GBytes  15.7 Gbits/sec

NFS Server, the CPU is Xeon E5-2680 v2. For client the CPU is Xeon CPU E7-8891 v3.

RDMA is not be set.

Well thats your probme there. Try with multiple transfers to see if its a single thread speed limit.

 

Try out using rdma, it should lower the cpu overhed.