Connecting windows and freenas p2p

[radmanhs]

I have been working on this server for the past 3 days in a row and I think I'm at my last 1 or 2 hurdles.... Right now I'm having the issue that my 2 systems won't connect via direct ethernet.  I have a gigabyte 10gb card in my main rig and an intel x550-t2 card in my freenas machine.  I set an ip address for both cards and adjusted some little things like mtu, but on the windows side it says "no network access".  I can transfer files across, but it goes through the 1gb network via a switch, and I can ping the nics ip address and get a response.

 

ip of windows) 192.168.200.2

ip of freenas) 192.168.200.1

both have /24 subnet

 

Any ideas?

[NelizMastr]

22 minutes ago, radmanhs said:

it goes through the 1gb network via a switch, and I can ping the nics ip address and get a response.

So you've mapped the drive on hostname rather than IP? If so, then you're dealing with Windows' wonderful NetBIOS resolving. Map the share to the IP of the freenas server on the 200 subnet.

 

The 200 subnet isn't actually your main network, right? It needs to be completely separate and unroutable. Just to confirm.

[radmanhs]

1 hour ago, NelizMastr said:

So you've mapped the drive on hostname rather than IP? If so, then you're dealing with Windows' wonderful NetBIOS resolving. Map the share to the IP of the freenas server on the 200 subnet.

 

The 200 subnet isn't actually your main network, right? It needs to be completely separate and unroutable. Just to confirm.

Seems like I have it working alright one way, but not the other.  With 6 wd reds i get between 200-450MB/s depending on if I'm moving large or small files.  However on the way back from from freenas to windows it is going through the same network as before and capping at 120MB/s.  

 

And yes, the 200 is a subnet just between the 2 machines. 

[radmanhs]

additionally, it may just be because of the files, but the speeds seem to be in-persistent at times.  I'm transferring my steam library to the server and speeds will be about 130MB/s for a little, then drop to sub 300KB/s for a while, then start jumping.

[NelizMastr]

56 minutes ago, radmanhs said:

 However on the way back from from freenas to windows it is going through the same network as before and capping at 120MB/s.  

 

You could verify whether that's the problem by disconnecting the other LAN port (so 10GbE only) and testing the transfer again. If speeds stay the same, it's down to the file size. Small files will really tank the transfer speed compared to a sustained transfer of large files.

[radmanhs]

21 hours ago, NelizMastr said:

You could verify whether that's the problem by disconnecting the other LAN port (so 10GbE only) and testing the transfer again. If speeds stay the same, it's down to the file size. Small files will really tank the transfer speed compared to a sustained transfer of large files.

I ran a crytsaldiskmark test on the nas with only the p2p subnet and peaked at 150MB/s read.  Writes are fine at 620MB/s peak.

 

Seems really slow compared to what I have seen with similar drives.

[NelizMastr]

3 minutes ago, radmanhs said:

I ran a crytsaldiskmark test on the nas with only the p2p subnet and peaked at 150MB/s read.  Writes are fine at 620MB/s peak.

 

Seems really slow compared to what I have seen with similar drives.

What's especially weird is the slow read performance. In RAID, it's supposed to be the other way around. Blazing reads, mediocre writes. 

[radmanhs]

Another quick run... with even faster writes.

 

I'm so confused.

[leadeater]

@Windows7ge

[radmanhs]

It wouldn't be the raid 1 ssd cache that is slowing it down, right?

 

Edit: Doesn't seem like it, this is an almost identical array without cache

 

 

[Windows7ge]

Under Services > SMB > Settings, did you make sure under Bind IP Addresses that your local network is not selected?

 

Also you setup a SSD cache for the volume? If the volume's only use is an SMB share then you should remove them from the configuration.

On ZFS SSD caching only accelerates synchronous reads/writes such as for database or virtual machines. As it stands FreeNAS and ZFS uses your RAM as a read cache for your most frequent files. Attempting to use an SSD would actually slow down the potential read performance.

 

A network share uses primarily asynchronous operations which is why a read/write cache is useless.

 

If the system is writing over the 192.168.200.0 network but reading over the local network that may have to do with the before mentioned tick box or it may have to do with the interfaces metric. If you set a static metric one that is lower than the local network that may help:

By default on FreeNAS's side the interface metric is 0.

 

The NIC is going to report "No Internet Access" or perhaps No Network Access but if you can ping the servers NIC then that error is irrelevant. it just means there's no Default Gateway and no Internet access.

 

Also if you have no plans on expanding the network you might as well put it on a /30 subnet.

 

As for the average performance seeming lower than other people are reporting, this is also an issue I deal with of which I have not yet resolved. The latest running theory is that SMB is just not well optimized on Linux/UNIX. With 10Gbit 1 user isn't really meant to have full utilization of it so the driver support just isn't really there. There are many tweaks we can go over if you want to try them but for me they didn't do much though others swear they helped.

[radmanhs]

3 hours ago, Windows7ge said:

 

Alright, I took a look at both, the bind addresses and the metric and they were already set with the subnet unchecked and automatic metric.  The tests were on an iscsi share so I had access to a "letter drive" rather than just a network drive.  But the performance was the same on an smb and iscsi share.  

 

As for the share I didn't think it could hurt, but if it is I can use it somewhere else.  Probably use it for VM's.

[Windows7ge]

31 minutes ago, radmanhs said:

Alright, I took a look at both, the bind addresses and the metric and they were already set with the subnet unchecked and automatic metric.  The tests were on an iscsi share so I had access to a "letter drive" rather than just a network drive.  But the performance was the same on an smb and iscsi share.  

 

As for the share I didn't think it could hurt, but if it is I can use it somewhere else.  Probably use it for VM's.

You can map a network folder to a network drive with a letter using SMB. iSCSI isn't worth it unless it yields some benefit or is really necessary.

[radmanhs]

29 minutes ago, Windows7ge said:

You can map a network folder to a network drive with a letter using SMB. iSCSI isn't worth it unless it yields some benefit or is really necessary.

Ahhh ok. That makes my life a bit easier. Now I just gotta figure out if I can boost my read speeds and why my plex quit working, but that's a topic for another thread 

[radmanhs]

I removed the ssd cache and made it it's own mirrored array for VMs and now I know for sure there is something up.  It has the exact same speeds for reads at 145 MB/s.  I have tried at least 8 different solutions with no change.

[leadeater]

15 hours ago, radmanhs said:

I removed the ssd cache and made it it's own mirrored array for VMs and now I know for sure there is something up.  It has the exact same speeds for reads at 145 MB/s.  I have tried at least 8 different solutions with no change.

What CPU is in the server?

[radmanhs]

8 hours ago, leadeater said:

What CPU is in the server?

dual xeon 2670's

[Windows7ge]

On 10/26/2018 at 3:07 AM, radmanhs said:

dual xeon 2670's

Is your original problem fixed?

What are you using to measure the read/write performance of the VMs?

[radmanhs]

1 hour ago, Windows7ge said:

Is your original problem fixed?

What are you using to measure the read/write performance of the VMs?

Ya, the original problem was solved. I tested everything with crystal disk and real life transfers, which came out to about the same speeds 

[Windows7ge]

4 minutes ago, radmanhs said:

Ya, the original problem was solved. I tested everything with crystal disk and real life transfers, which came out to about the same speeds 

What is the exact copy you tried? Was it from within a VM to the array? I'm a bit confused as to the from and to the transfer is taking place.

[radmanhs]

49 minutes ago, Windows7ge said:

What is the exact copy you tried? Was it from within a VM to the array? I'm a bit confused as to the from and to the transfer is taking place.

It was just a transfer from my main pc to the mirror through a smb share.  I wasn't running a VM off them yet, just going to use it for VMs in the future

[Windows7ge]

9 hours ago, radmanhs said:

It was just a transfer from my main pc to the mirror through a smb share.  I wasn't running a VM off them yet, just going to use it for VMs in the future

Ah, that makes sense. I've noticed that FreeNAS is like that. It doesn't really respond to SSDs like you'd expect a server to. There's some things that work against you like ZFS in itself is not optimized to be the fastest file system rather data integrity is it's primary objective. Trying out BTRFS might yield better results.

 

However. Essentially 150MB/s is definitely lower than what you should be seeing. When it comes to read performance the E5-2670 really doesn't do 10Gbit justice. If you haven't already enable Jumbo Packets. It helps the CPU along by making it do less operations when transmitting packets across the network.

 

Do you know how to enable Jumbo Packets on Windows & FreeNAS?

[radmanhs]

On 10/28/2018 at 8:16 AM, Windows7ge said:

Do you know how to enable Jumbo Packets on Windows & FreeNAS?

Yep, Jumbo frames are already enabled on both systems.  Unless something has changed since I saw what to enable windows is on 9014 bytes and the nic in freenas in the options tab is set to mtu 9000.

[Windows7ge]

52 minutes ago, radmanhs said:

Yep, Jumbo frames are already enabled on both systems.  Unless something has changed since I saw what to enable windows is on 9014 bytes and the nic in freenas in the options tab is set to mtu 9000.

Other things you can try doing are editing the SMB parameters.

Under Auxiliary parameters input:

strict allocate = No
read raw = Yes
write raw = Yes
server signing = No
strict locking = No
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
min receivefile size = 16384
use sendfile = Yes
aio read size = 16384
aio write size = 16384

You should be able to copy & paste it into the menu. See what that does.

You can also set a series of tunables which helps optimize the system for 10Gbit:

sysctl kern.ipc.maxsockbuf=16777216
sysctl net.inet.ip.intr_queue_maxlen=2048
sysctl net.inet.tcp.recvbuf_inc=524288
sysctl net.inet.tcp.recvbuf_max=16777216
sysctl net.inet.tcp.recvspace=4194304
sysctl net.inet.tcp.sendbuf_inc=32768
sysctl net.inet.tcp.sendbuf_max=16777216
sysctl net.inet.tcp.sendspace=2097152
sysctl net.route.netisr_maxqlen=2048

I have to recommend that you backup your sysconfig before touching the tuneables. Misconfiguring a tunable can completely lock you out of the WebUI, SSH, & your network share leaving only the console menu on the server itself accessible. Keep a backup both off the server & on the server in a dataset.

 

In the event you lockout the server due to a bad config you can reload from a previous file. Name the file something easy but keep .db on the end. From shell navigate to the dataset directory containing the config file and run:

cp file.db /mnt/data/freenas-v1.db && reboot

This will replace the servers current configuration with the one it was running previously it will then restart the server for you and you should be up and running again. Make a habit of backing up this file before you make changes to the server and after you verify the new change hasn't harmed anything. If you keep your volumes encrypted make sure you keep a copy of the keys off the server too so you can recover them in the event of total failure of the OS.

[radmanhs]

7 hours ago, Windows7ge said:

Other things you can try doing are editing the SMB parameters.

Under Auxiliary parameters input:

strict allocate = No
read raw = Yes
write raw = Yes
server signing = No
strict locking = No
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
min receivefile size = 16384
use sendfile = Yes
aio read size = 16384
aio write size = 16384

You should be able to copy & paste it into the menu. See what that does.

You can also set a series of tunables which helps optimize the system for 10Gbit:

sysctl kern.ipc.maxsockbuf=16777216
sysctl net.inet.ip.intr_queue_maxlen=2048
sysctl net.inet.tcp.recvbuf_inc=524288
sysctl net.inet.tcp.recvbuf_max=16777216
sysctl net.inet.tcp.recvspace=4194304
sysctl net.inet.tcp.sendbuf_inc=32768
sysctl net.inet.tcp.sendbuf_max=16777216
sysctl net.inet.tcp.sendspace=2097152
sysctl net.route.netisr_maxqlen=2048

I have to recommend that you backup your sysconfig before touching the tuneables. Misconfiguring a tunable can completely lock you out of the WebUI, SSH, & your network share leaving only the console menu on the server itself accessible. Keep a backup both off the server & on the server in a dataset.

 

In the event you lockout the server due to a bad config you can reload from a previous file. Name the file something easy but keep .db on the end. From shell navigate to the dataset directory containing the config file and run:

cp file.db /mnt/data/freenas-v1.db && reboot

This will replace the servers current configuration with the one it was running previously it will then restart the server for you and you should be up and running again. Make a habit of backing up this file before you make changes to the server and after you verify the new change hasn't harmed anything. If you keep your volumes encrypted make sure you keep a copy of the keys off the server too so you can recover them in the event of total failure of the OS.

the parameter changed the read a tiny bit, but pretty negligible.  For the tunables.  Do I have to set each one individually? Or put them all in the value together.  Additionally, what do I put the variables as?  Or do you put each tunable as the variable?  Finally, I'm assuming you change the type to sysctl?