Diagnosing a HD or Network Bottleneck on 10G Network

[UserSN]

Hello Everyone,

 

I'm having a tough time pinpointing my bottleneck. I'm trying to setup a fileserver using FreeNAS and ISCSi some SSD's on a vdev into some Hyper-V VM's on different physical machines. I bought a dedicated cisco switch for all this traffic, ended up buying cat8 cables to rule out any cabling issue and i'm still at transfer speeds between 100-150mb/sec using SAMBA.

 

I've ruled out the switch being the issue by connecting two of my nodes (windows boxes) together and running some iperf tests between the two nodes and my speed problem is still there so I don't think the switch is my problem area.
Initially after getting everything up and running I used ATTO Disk benchmark to test the reads/writes from NODE1 to FreeNAS BOX and the results were:

 

1) Testing against the internal Raid 10 Array on the physical box, Node1

32MB | 756 MB/s Write | 1.43 GB/s Read

48MB | 834 MB/s Write | 1.52 GB/s Read

64MB | 908 MB/s Write | 1.36 GB/s Read

2) Testing against a 7200RPM vdev through ISCI on the Freenas box from Node1

32MB | 1.04 GB/s Write | 474 MB/s Read

48MB | 1.01 GB/s Write | 521 MB/s Read

64MB | 1.00 GB/s Write | 449 MB/s Read

 

3) Testing against a WD Red SSD vdev through ISCI on the Freenas box from Node1

32MB | 1.10 GB/s Write | 456 MB/s Read

48MB | 1.10 GB/s Write | 464 MB/s Read

64MB | 1.07 GB/s Write | 485 MB/s Read

 

These results were weird to me as so I downloaded Iperf3 and ran some tests from Node's 1 & 2 and I generally average out to 2-3 GBits/s to the FreeNAS box. (Node -> Switch -> Freenas)
Then I tested the transfer rate from Node1 to Node2 and the same average between 1-2 GBits/s. Changing up the argument's sometimes the averages were 1gbit across the tests other times it varied between 1-3. (Node1 -> Switch -> Node2)

Then I connected Node1 & Node2 directly to each-other and ran the same tests and got the same results. (Node1 -> Node2)

 

I have support cases open with Intel, Cisco & Microsoft but noone's given me any concrete direction to explore in 2 weeks, they've only asked for countless logs and other test info. I ran the machine to machine test direct so I can at least eliminate the switch as the problem. Since I'm consistently getting capped at 3gbits/s I'm thinking it could be my drives as these Node tests are not running SSD's but I'm hoping for some feedback from the community. Reading through this article it seems I'm having a similar issue. 

 

Any thoughts, ideas, other tests to run would be extremely helpful, thanks and sorry for the expansive post.

 

[UserSN]

Also, forgot to mention all these tests were on the physical machines, I also ran tests from VM's in hyper-V and over there it's even worse I'm not passing gigabit speeds ever, even though the status on adapter shows it running 10G speed. 

[Electronics Wizardy]

3 minutes ago, UserSN said:

Then I tested the transfer rate from Node1 to Node2 and the same average between 1-2 GBits/s. Changing up the argument's sometimes the averages were 1gbit across the tests other times it varied between 1-3. (Node1 -> Switch -> Node2)

Then I connected Node1 & Node2 directly to each-other and ran the same tests and got the same results. (Node1 -> Node2)

 

If those are your iperf speeds, then thats a network issue. Look there more. The disk performance seems to be much faster than the network speed.

 

You can try setting the -P to get multiple streams at once.

 

 

[UserSN]

6 minutes ago, Electronics Wizardy said:

If those are your iperf speeds, then thats a network issue. Look there more. The disk performance seems to be much faster than the network speed.

 

You can try setting the -P to get multiple streams at once.

 

 

Here's a test running -P 16

PS C:\Users\Administrator\Desktop\iperf-3.1.3-win64\iperf-3.1.3-win64> ./iperf3.exe -c 192.168.69.20 -P 16
Connecting to host 192.168.69.20, port 5201
[  4] local 192.168.69.11 port 52809 connected to 192.168.69.20 port 5201
[  6] local 192.168.69.11 port 52810 connected to 192.168.69.20 port 5201
[  8] local 192.168.69.11 port 52811 connected to 192.168.69.20 port 5201
[ 10] local 192.168.69.11 port 52812 connected to 192.168.69.20 port 5201
[ 12] local 192.168.69.11 port 52813 connected to 192.168.69.20 port 5201
[ 14] local 192.168.69.11 port 52814 connected to 192.168.69.20 port 5201
[ 16] local 192.168.69.11 port 52815 connected to 192.168.69.20 port 5201
[ 18] local 192.168.69.11 port 52816 connected to 192.168.69.20 port 5201
[ 20] local 192.168.69.11 port 52817 connected to 192.168.69.20 port 5201
[ 22] local 192.168.69.11 port 52818 connected to 192.168.69.20 port 5201
[ 24] local 192.168.69.11 port 52819 connected to 192.168.69.20 port 5201
[ 26] local 192.168.69.11 port 52820 connected to 192.168.69.20 port 5201
[ 28] local 192.168.69.11 port 52821 connected to 192.168.69.20 port 5201
[ 30] local 192.168.69.11 port 52822 connected to 192.168.69.20 port 5201
[ 32] local 192.168.69.11 port 52823 connected to 192.168.69.20 port 5201
[ 34] local 192.168.69.11 port 52824 connected to 192.168.69.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.02   sec  33.9 MBytes   277 Mbits/sec
[  6]   0.00-1.02   sec  34.1 MBytes   279 Mbits/sec
[  8]   0.00-1.02   sec  33.9 MBytes   277 Mbits/sec
[ 10]   0.00-1.02   sec  34.0 MBytes   278 Mbits/sec
[ 12]   0.00-1.02   sec  34.1 MBytes   279 Mbits/sec
[ 14]   0.00-1.02   sec  34.1 MBytes   279 Mbits/sec
[ 16]   0.00-1.02   sec  34.1 MBytes   279 Mbits/sec
[ 18]   0.00-1.02   sec  33.9 MBytes   277 Mbits/sec
[ 20]   0.00-1.02   sec  34.0 MBytes   278 Mbits/sec
[ 22]   0.00-1.02   sec  34.0 MBytes   278 Mbits/sec
[ 24]   0.00-1.02   sec  34.1 MBytes   279 Mbits/sec
[ 26]   0.00-1.02   sec  34.1 MBytes   279 Mbits/sec
[ 28]   0.00-1.02   sec  33.9 MBytes   277 Mbits/sec
[ 30]   0.00-1.02   sec  34.0 MBytes   278 Mbits/sec
[ 32]   0.00-1.02   sec  34.0 MBytes   278 Mbits/sec
[ 34]   0.00-1.02   sec  34.0 MBytes   278 Mbits/sec
[SUM]   0.00-1.02   sec   544 MBytes  4.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.02-2.00   sec  37.4 MBytes   321 Mbits/sec
[  6]   1.02-2.00   sec  37.4 MBytes   321 Mbits/sec
[  8]   1.02-2.00   sec  37.4 MBytes   321 Mbits/sec
[ 10]   1.02-2.00   sec  37.1 MBytes   318 Mbits/sec
[ 12]   1.02-2.00   sec  37.2 MBytes   320 Mbits/sec
[ 14]   1.02-2.00   sec  37.2 MBytes   320 Mbits/sec
[ 16]   1.02-2.00   sec  37.1 MBytes   318 Mbits/sec
[ 18]   1.02-2.00   sec  37.2 MBytes   320 Mbits/sec
[ 20]   1.02-2.00   sec  37.1 MBytes   318 Mbits/sec
[ 22]   1.02-2.00   sec  37.1 MBytes   318 Mbits/sec
[ 24]   1.02-2.00   sec  37.0 MBytes   317 Mbits/sec
[ 26]   1.02-2.00   sec  37.1 MBytes   318 Mbits/sec
[ 28]   1.02-2.00   sec  37.0 MBytes   317 Mbits/sec
[ 30]   1.02-2.00   sec  36.6 MBytes   314 Mbits/sec
[ 32]   1.02-2.00   sec  36.1 MBytes   310 Mbits/sec
[ 34]   1.02-2.00   sec  36.6 MBytes   314 Mbits/sec
[SUM]   1.02-2.00   sec   593 MBytes  5.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  41.0 MBytes   345 Mbits/sec
[  6]   2.00-3.00   sec  40.9 MBytes   344 Mbits/sec
[  8]   2.00-3.00   sec  40.9 MBytes   344 Mbits/sec
[ 10]   2.00-3.00   sec  40.4 MBytes   339 Mbits/sec
[ 12]   2.00-3.00   sec  40.8 MBytes   343 Mbits/sec
[ 14]   2.00-3.00   sec  40.9 MBytes   344 Mbits/sec
[ 16]   2.00-3.00   sec  40.8 MBytes   343 Mbits/sec
[ 18]   2.00-3.00   sec  40.4 MBytes   339 Mbits/sec
[ 20]   2.00-3.00   sec  40.1 MBytes   337 Mbits/sec
[ 22]   2.00-3.00   sec  40.2 MBytes   338 Mbits/sec
[ 24]   2.00-3.00   sec  40.2 MBytes   338 Mbits/sec
[ 26]   2.00-3.00   sec  40.2 MBytes   338 Mbits/sec
[ 28]   2.00-3.00   sec  40.0 MBytes   336 Mbits/sec
[ 30]   2.00-3.00   sec  40.0 MBytes   336 Mbits/sec
[ 32]   2.00-3.00   sec  40.0 MBytes   336 Mbits/sec
[ 34]   2.00-3.00   sec  39.9 MBytes   335 Mbits/sec
[SUM]   2.00-3.00   sec   647 MBytes  5.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.01   sec  45.5 MBytes   378 Mbits/sec
[  6]   3.00-4.01   sec  45.5 MBytes   378 Mbits/sec
[  8]   3.00-4.01   sec  44.2 MBytes   368 Mbits/sec
[ 10]   3.00-4.01   sec  44.9 MBytes   373 Mbits/sec
[ 12]   3.00-4.01   sec  45.9 MBytes   381 Mbits/sec
[ 14]   3.00-4.01   sec  45.9 MBytes   381 Mbits/sec
[ 16]   3.00-4.01   sec  46.1 MBytes   383 Mbits/sec
[ 18]   3.00-4.01   sec  45.1 MBytes   375 Mbits/sec
[ 20]   3.00-4.01   sec  44.1 MBytes   367 Mbits/sec
[ 22]   3.00-4.01   sec  45.9 MBytes   381 Mbits/sec
[ 24]   3.00-4.01   sec  45.5 MBytes   378 Mbits/sec
[ 26]   3.00-4.01   sec  45.2 MBytes   376 Mbits/sec
[ 28]   3.00-4.01   sec  45.5 MBytes   378 Mbits/sec
[ 30]   3.00-4.01   sec  45.0 MBytes   374 Mbits/sec
[ 32]   3.00-4.01   sec  45.1 MBytes   375 Mbits/sec
[ 34]   3.00-4.01   sec  45.4 MBytes   377 Mbits/sec
[SUM]   3.00-4.01   sec   725 MBytes  6.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.01-5.01   sec  46.6 MBytes   391 Mbits/sec
[  6]   4.01-5.01   sec  47.0 MBytes   394 Mbits/sec
[  8]   4.01-5.01   sec  46.5 MBytes   390 Mbits/sec
[ 10]   4.01-5.01   sec  46.8 MBytes   392 Mbits/sec
[ 12]   4.01-5.01   sec  46.4 MBytes   389 Mbits/sec
[ 14]   4.01-5.01   sec  47.0 MBytes   394 Mbits/sec
[ 16]   4.01-5.01   sec  47.2 MBytes   397 Mbits/sec
[ 18]   4.01-5.01   sec  46.5 MBytes   390 Mbits/sec
[ 20]   4.01-5.01   sec  46.6 MBytes   391 Mbits/sec
[ 22]   4.01-5.01   sec  46.4 MBytes   389 Mbits/sec
[ 24]   4.01-5.01   sec  46.2 MBytes   388 Mbits/sec
[ 26]   4.01-5.01   sec  46.9 MBytes   393 Mbits/sec
[ 28]   4.01-5.01   sec  46.6 MBytes   391 Mbits/sec
[ 30]   4.01-5.01   sec  46.5 MBytes   390 Mbits/sec
[ 32]   4.01-5.01   sec  45.5 MBytes   382 Mbits/sec
[ 34]   4.01-5.01   sec  46.4 MBytes   389 Mbits/sec
[SUM]   4.01-5.01   sec   745 MBytes  6.25 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.01-6.00   sec  46.4 MBytes   391 Mbits/sec
[  6]   5.01-6.00   sec  46.4 MBytes   391 Mbits/sec
[  8]   5.01-6.00   sec  46.0 MBytes   388 Mbits/sec
[ 10]   5.01-6.00   sec  45.5 MBytes   384 Mbits/sec
[ 12]   5.01-6.00   sec  46.0 MBytes   388 Mbits/sec
[ 14]   5.01-6.00   sec  46.1 MBytes   389 Mbits/sec
[ 16]   5.01-6.00   sec  46.5 MBytes   392 Mbits/sec
[ 18]   5.01-6.00   sec  45.8 MBytes   386 Mbits/sec
[ 20]   5.01-6.00   sec  45.8 MBytes   386 Mbits/sec
[ 22]   5.01-6.00   sec  46.2 MBytes   390 Mbits/sec
[ 24]   5.01-6.00   sec  45.9 MBytes   387 Mbits/sec
[ 26]   5.01-6.00   sec  45.8 MBytes   386 Mbits/sec
[ 28]   5.01-6.00   sec  45.9 MBytes   387 Mbits/sec
[ 30]   5.01-6.00   sec  45.2 MBytes   382 Mbits/sec
[ 32]   5.01-6.00   sec  45.5 MBytes   384 Mbits/sec
[ 34]   5.01-6.00   sec  45.9 MBytes   387 Mbits/sec
[SUM]   5.01-6.00   sec   735 MBytes  6.20 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  48.6 MBytes   410 Mbits/sec
[  6]   6.00-7.00   sec  48.9 MBytes   412 Mbits/sec
[  8]   6.00-7.00   sec  48.4 MBytes   408 Mbits/sec
[ 10]   6.00-7.00   sec  48.0 MBytes   404 Mbits/sec
[ 12]   6.00-7.00   sec  48.6 MBytes   410 Mbits/sec
[ 14]   6.00-7.00   sec  48.8 MBytes   411 Mbits/sec
[ 16]   6.00-7.00   sec  48.4 MBytes   408 Mbits/sec
[ 18]   6.00-7.00   sec  48.4 MBytes   408 Mbits/sec
[ 20]   6.00-7.00   sec  47.9 MBytes   403 Mbits/sec
[ 22]   6.00-7.00   sec  48.5 MBytes   409 Mbits/sec
[ 24]   6.00-7.00   sec  48.0 MBytes   404 Mbits/sec
[ 26]   6.00-7.00   sec  48.4 MBytes   408 Mbits/sec
[ 28]   6.00-7.00   sec  47.8 MBytes   402 Mbits/sec
[ 30]   6.00-7.00   sec  47.4 MBytes   399 Mbits/sec
[ 32]   6.00-7.00   sec  48.1 MBytes   405 Mbits/sec
[ 34]   6.00-7.00   sec  47.5 MBytes   400 Mbits/sec
[SUM]   6.00-7.00   sec   772 MBytes  6.50 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  45.5 MBytes   382 Mbits/sec
[  6]   7.00-8.00   sec  45.8 MBytes   384 Mbits/sec
[  8]   7.00-8.00   sec  45.9 MBytes   385 Mbits/sec
[ 10]   7.00-8.00   sec  45.8 MBytes   384 Mbits/sec
[ 12]   7.00-8.00   sec  46.1 MBytes   387 Mbits/sec
[ 14]   7.00-8.00   sec  46.0 MBytes   386 Mbits/sec
[ 16]   7.00-8.00   sec  45.6 MBytes   383 Mbits/sec
[ 18]   7.00-8.00   sec  45.8 MBytes   384 Mbits/sec
[ 20]   7.00-8.00   sec  45.2 MBytes   380 Mbits/sec
[ 22]   7.00-8.00   sec  45.8 MBytes   384 Mbits/sec
[ 24]   7.00-8.00   sec  45.1 MBytes   379 Mbits/sec
[ 26]   7.00-8.00   sec  45.6 MBytes   383 Mbits/sec
[ 28]   7.00-8.00   sec  45.4 MBytes   381 Mbits/sec
[ 30]   7.00-8.00   sec  45.2 MBytes   380 Mbits/sec
[ 32]   7.00-8.00   sec  45.1 MBytes   379 Mbits/sec
[ 34]   7.00-8.00   sec  44.9 MBytes   376 Mbits/sec
[SUM]   7.00-8.00   sec   729 MBytes  6.11 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.01   sec  49.0 MBytes   407 Mbits/sec
[  6]   8.00-9.01   sec  49.0 MBytes   407 Mbits/sec
[  8]   8.00-9.01   sec  48.8 MBytes   405 Mbits/sec
[ 10]   8.00-9.01   sec  48.5 MBytes   403 Mbits/sec
[ 12]   8.00-9.01   sec  48.9 MBytes   406 Mbits/sec
[ 14]   8.00-9.01   sec  49.0 MBytes   407 Mbits/sec
[ 16]   8.00-9.01   sec  48.6 MBytes   404 Mbits/sec
[ 18]   8.00-9.01   sec  48.9 MBytes   406 Mbits/sec
[ 20]   8.00-9.01   sec  48.5 MBytes   403 Mbits/sec
[ 22]   8.00-9.01   sec  48.5 MBytes   403 Mbits/sec
[ 24]   8.00-9.01   sec  48.6 MBytes   404 Mbits/sec
[ 26]   8.00-9.01   sec  48.9 MBytes   406 Mbits/sec
[ 28]   8.00-9.01   sec  48.5 MBytes   403 Mbits/sec
[ 30]   8.00-9.01   sec  47.5 MBytes   395 Mbits/sec
[ 32]   8.00-9.01   sec  48.2 MBytes   401 Mbits/sec
[ 34]   8.00-9.01   sec  48.2 MBytes   401 Mbits/sec
[SUM]   8.00-9.01   sec   778 MBytes  6.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.01-10.00  sec  41.2 MBytes   349 Mbits/sec
[  6]   9.01-10.00  sec  41.5 MBytes   351 Mbits/sec
[  8]   9.01-10.00  sec  41.2 MBytes   349 Mbits/sec
[ 10]   9.01-10.00  sec  41.1 MBytes   348 Mbits/sec
[ 12]   9.01-10.00  sec  41.8 MBytes   354 Mbits/sec
[ 14]   9.01-10.00  sec  41.6 MBytes   352 Mbits/sec
[ 16]   9.01-10.00  sec  41.4 MBytes   350 Mbits/sec
[ 18]   9.01-10.00  sec  41.5 MBytes   351 Mbits/sec
[ 20]   9.01-10.00  sec  41.1 MBytes   348 Mbits/sec
[ 22]   9.01-10.00  sec  41.0 MBytes   347 Mbits/sec
[ 24]   9.01-10.00  sec  40.5 MBytes   343 Mbits/sec
[ 26]   9.01-10.00  sec  41.4 MBytes   350 Mbits/sec
[ 28]   9.01-10.00  sec  41.0 MBytes   347 Mbits/sec
[ 30]   9.01-10.00  sec  40.8 MBytes   345 Mbits/sec
[ 32]   9.01-10.00  sec  41.2 MBytes   349 Mbits/sec
[ 34]   9.01-10.00  sec  40.9 MBytes   346 Mbits/sec
[SUM]   9.01-10.00  sec   659 MBytes  5.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   435 MBytes   365 Mbits/sec                  sender
[  4]   0.00-10.00  sec   435 MBytes   365 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   436 MBytes   366 Mbits/sec                  sender
[  6]   0.00-10.00  sec   436 MBytes   366 Mbits/sec                  receiver
[  8]   0.00-10.00  sec   433 MBytes   363 Mbits/sec                  sender
[  8]   0.00-10.00  sec   433 MBytes   363 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec   432 MBytes   362 Mbits/sec                  sender
[ 10]   0.00-10.00  sec   432 MBytes   362 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec   436 MBytes   366 Mbits/sec                  sender
[ 12]   0.00-10.00  sec   435 MBytes   365 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec   437 MBytes   366 Mbits/sec                  sender
[ 14]   0.00-10.00  sec   436 MBytes   366 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec   436 MBytes   366 Mbits/sec                  sender
[ 16]   0.00-10.00  sec   436 MBytes   365 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec   433 MBytes   364 Mbits/sec                  sender
[ 18]   0.00-10.00  sec   433 MBytes   363 Mbits/sec                  receiver
[ 20]   0.00-10.00  sec   430 MBytes   361 Mbits/sec                  sender
[ 20]   0.00-10.00  sec   430 MBytes   361 Mbits/sec                  receiver
[ 22]   0.00-10.00  sec   434 MBytes   364 Mbits/sec                  sender
[ 22]   0.00-10.00  sec   433 MBytes   364 Mbits/sec                  receiver
[ 24]   0.00-10.00  sec   431 MBytes   362 Mbits/sec                  sender
[ 24]   0.00-10.00  sec   431 MBytes   361 Mbits/sec                  receiver
[ 26]   0.00-10.00  sec   434 MBytes   364 Mbits/sec                  sender
[ 26]   0.00-10.00  sec   433 MBytes   363 Mbits/sec                  receiver
[ 28]   0.00-10.00  sec   432 MBytes   362 Mbits/sec                  sender
[ 28]   0.00-10.00  sec   431 MBytes   362 Mbits/sec                  receiver
[ 30]   0.00-10.00  sec   428 MBytes   359 Mbits/sec                  sender
[ 30]   0.00-10.00  sec   428 MBytes   359 Mbits/sec                  receiver
[ 32]   0.00-10.00  sec   429 MBytes   360 Mbits/sec                  sender
[ 32]   0.00-10.00  sec   429 MBytes   360 Mbits/sec                  receiver
[ 34]   0.00-10.00  sec   430 MBytes   360 Mbits/sec                  sender
[ 34]   0.00-10.00  sec   429 MBytes   360 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  6.76 GBytes  5.81 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  6.76 GBytes  5.80 Gbits/sec                  receiver

[UserSN]

PS C:\Users\Administrator\Desktop\iperf-3.1.3-win64\iperf-3.1.3-win64> ./iperf3.exe -c 192.168.69.20 -w 128m
Connecting to host 192.168.69.20, port 5201
[  4] local 192.168.69.11 port 52847 connected to 192.168.69.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   455 MBytes  3.82 Gbits/sec
[  4]   1.00-2.00   sec   191 MBytes  1.60 Gbits/sec
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.00-4.00   sec   119 MBytes  1.00 Gbits/sec
[  4]   4.00-5.00   sec   310 MBytes  2.60 Gbits/sec
[  4]   5.00-6.00   sec  21.2 MBytes   178 Mbits/sec
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   7.00-8.00   sec   128 MBytes  1.07 Gbits/sec
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.00-10.00  sec   128 MBytes  1.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.32 GBytes  1.13 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.00 GBytes   859 Mbits/sec                  receiver

iperf Done.

[Electronics Wizardy]

9 minutes ago, UserSN said:

PS C:\Users\Administrator\Desktop\iperf-3.1.3-win64\iperf-3.1.3-win64> ./iperf3.exe -c 192.168.69.20 -w 128m
Connecting to host 192.168.69.20, port 5201
[  4] local 192.168.69.11 port 52847 connected to 192.168.69.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   455 MBytes  3.82 Gbits/sec
[  4]   1.00-2.00   sec   191 MBytes  1.60 Gbits/sec
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.00-4.00   sec   119 MBytes  1.00 Gbits/sec
[  4]   4.00-5.00   sec   310 MBytes  2.60 Gbits/sec
[  4]   5.00-6.00   sec  21.2 MBytes   178 Mbits/sec
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   7.00-8.00   sec   128 MBytes  1.07 Gbits/sec
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.00-10.00  sec   128 MBytes  1.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.32 GBytes  1.13 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.00 GBytes   859 Mbits/sec                  receiver

iperf Done.

A bit better on the other ones, but not gret.

 

Try removing your switch nd doing direct connections.

[UserSN]

 

Just now, Electronics Wizardy said:

A bit better on the other ones, but not gret.

 

Try removing your switch nd doing direct connections.

this is directly connected with no switch

[Dutch_Master]

Keep in mind the difference between bits (b) and bytes (B). So a 10Gbit connection is not the same as a 10GB connection.

 

As for drives being a bottleneck, only NVMe drives can handle large speeds, SATA is capped at 600MB/s theoretically (in practice more like 550MB/s) while NVMe on PCIe gen 3 can get you 3.5 GB/s pratical, PCIe gen 4 doubles that.

[UserSN]

29 minutes ago, zhnu said:

What's the cpu load like during transfers?

Minimal barely 10%

[UserSN]

7 hours ago, Dutch_Master said:

Keep in mind the difference between bits (b) and bytes (B). So a 10Gbit connection is not the same as a 10GB connection.

 

As for drives being a bottleneck, only NVMe drives can handle large speeds, SATA is capped at 600MB/s theoretically (in practice more like 550MB/s) while NVMe on PCIe gen 3 can get you 3.5 GB/s pratical, PCIe gen 4 doubles that.

Right I understand that but i'm expecting atleast to saturate the read/write capacity of my SATA SSD's, on a file transfer I'm clocking between 100mb/s - 150mb/s nothing above that. On iperf it shows in mbits but I've done the conversion and I should be getting roughly 500mb/s (5gbits/s) transfers which I'm not, I'm consistently getting between 1-2.5 gbits with drops down to 500mbits at times.

[UserSN]

1 hour ago, zhnu said:

No errors on logs or dmesg? And on the Hyper-V nodes do you have VMQ enabled and is the virtual switch configured correctly?

These tests are on the host machines not the VM's, but i do have VMQ enabled and the VM tests are not going past gigabit speeds. No errors on the logs either.

[Defconed]

Hypothesis:

CPU is fully loaded on 1 core, so single thread-IO is capped by CPU. 10% util on 4x E7 can roughly translate to 40% for a single CPU. iperf with 16threads does not translate into single thread IO perf.

Suggestion:

Check single core load when performing single-thread testing, try doing iperf on just -p 1 and see if it works just like file IO. 

PS:

1.Did you properly do the grounding on both ends for your Cat8 cable? if not this may contribute negatively. (See if both the NIC ports and the Eth jack has a piece of metal that should have contact when connected, if you don't see it on both ends you probably shouldn't use that cable.) You can buy affordable direct-attached SPF+ 10G modules with optic cables built into it. Downside is that is has limited length and choices, upside is that there's no optics connector and the module is cheap(it's used, of course)

2.I remembered something called RDMA for NIC, which should be turned on if you need 10G or 40Gig-ish, as their bandwidth could bring a CPU core to its knees. Are you sure this is configured?

Edited November 25, 2020 by Defconed

[UserSN]

Hey Guys,

 

The ATTO Benchmark test's I ran were on a hyper-V VM, I think that gave me a false reading. I ended up running the same tests on the host's to the C drive in Raid 10 and i'm getting 150/write 200/read which matches the speeds i'm getting out of iperf. I think it's my 7200 RPM disks on the Nodes that's the bottleneck.