[Build-Log] FreeNAS "Boss-NAS" by FloRolf [Update 14th January 2019]

[FloRolf]

Hello, Guten Tag and Bonjour everyone,  

 

 

This is gonna be my "build log" and experience setting up my first NAS, which will be 
a custom build FreeNAS Box.

 

First of all, sorry for everyone (but 2 guys) who voted on my most recent topic
about the name of the NAS.
Everyone was highly into "BanaNAS" but i don't actually like it too much, lol.
As the title already suggests, i went with "Boss-NAS" @mr moose. It has its origin in the Star Wars
universe, where Boss Nass is the leader of the Gungans, who fight off the droid attack on Naboo

during the Phantom Menace.
(But what about the droid attack on the wookiees?)

 

This guy right here:

Spoiler

 

 

Here's the overview over the components that will be used:

 

CPU: AMD Ryzen R5 2400G
Cooler: Noctua NH-D14
Motherboard: Asus Prime X370-Pro
RAM: 16gb G.Skill Aegis 3000mhz
PSU: Seasonic Focus Plus Platinum 550w
Case: Fractal Design R5
NIC: Emulex Dual 10Gbit SFP+
Storage:
6x 4TB WD Red in Raidz2
1x 250gb Samsung 970Evo as Cache
2x 500gb Samsung 860Evo in Raid0 for high speed
2x Transcend 32gb USB Stick mirrored for the OS

 

Color Code:

Red=Don't have that yet or only partially

Orange=Not sure if it works in that config

Black=Already had those parts

Green=Already aquired

 

So as you can see, the thing will be pretty OP for most, as well as my own, usecases.
The quad core R5 with onboard graphics is good enough for media streaming up to 4k
and should also hold up nicely for any kind of VM testing.

 

FreeNAS recommends 1gb RAM for each TB of Storage but i'm not really sure if they are saying
1TB of actual storage or theoretical storage. The issue here is, that Raidz2 will basically eliminate
9TB of my storage (15TB of 24TB usable) for redundancy, making it possible for me to lose 2 Drives at once while still having
all my Data safe. I've read though, that the 1gb/1tb is actually not all that relevant.

 

The 550w PSU is Platinum, rated and should help to always keep my within the range of optimal efficiency
however i would've liked to get an even lower wattage unit but apparently those are very rare in ATX Form Factor
and 80+Platinum efficiency.

 

Now lets jump into the usecases of this NAS:

 

1. Backup
-> The NAS is ofc a Backup solution for all my critical and private Data such as my GF's Uni Stuff and ofc lots of Pictures.
I also want to configure the NAS to automatically backup all my phones pictures as soon as i enter my (or any?) wifi, the same
way google pictures does it essentially.

2. Game Storage
-> While i've read that some games may take several minutes to start from a NAS, i still want to give it a shot.
Considering i will be running 10Gbit to each of my machines, this should in theory work just fine.
If it turns out things are unplayable, i'll still keep the Games on the NAS so i can quickly grab them
from any other PC and just install them.

3. Cloud Storage
-> There are some plugins on FreeNAS which allow me to use the NAS outside of my Home Network so i can access
all my data at any time and at any place. This is a lower priority though as i first have to figure some other things out.

4. Streaming
-> I'm actually a huge fan of Blurays and 4k Blurays but i still want to move all my physical discs on to the NAS
in 100% of the quality. Each Bluray will therefore use around 30-50gb of space. Jikes. 
I've narrowed it down to Plex as my media server because it seems to be the most simple to use and has a wide variety
of compatibility.

5. VMs
-> Actually i've never had any interest in VMs but i wanna give it a try, maybe to learn some better about
Ubuntu or whatever. This is also a low priority but still interesting.

 

 

Next up the progress i'm currently finding myself in:

 

here are a few pics of the components i'm using

First off the Drives i already purchased. Very beautiful! Really enjoyed opening these puppies up! :3

Although, i sadly can not use the HDDs currently because FreeNAS, as of right now, doesnt allow for the Raid Array

to be expanded. That means i'll have to get everything installed at once and will not be able to expand in the future.

(Expanding is possible however, just not this particular Raid Array)

The two SATA SSDs on the other hand can be implemented already as you'll see a few pics down.

Spoiler

 

Next there are my two USB Drives that i'm using for the OS, FreeNAS. It was actually a pain in the arse

to get these as i've read so many reviews on Amazon about USB Drives and legit all of them are shiet.

Either they break after 1 day, their speeds are only 5mb/s or they overheat like crazy.

These ones got the best reviews so i chose them. They aer 32gb which is massivley oversized

but they were only 2€ more than the 16gb so whatever, lol.

They also have small heat issues when running the system, so does anyone have an idea to keep them

a little cooler? I was thinking about just glueing small vram heatsinks on them.

Spoiler

 

Here's the whole horror assembled right now, lmao. As i don't have a case yet and i really desperately wanted

to try everything out, i'm finding myself in this, not really safe, situation. Please don't do that kids.

You can see the Nvme and Sata SSD's "Installed" Though as well as the massive NH-D14 cooler and the PSU.

There's also the motherboard, the Asus Prime X370-Pro which was one of only 2 (i think) AM4 Boards i could find that feature

8 Sata ports. It's almost impossible. I really really hope that all SATA ports will still be active when the Nvme is installed.

Some boards will turn off 2 SATA ports for that but i haven't found anything about that in the manual, which you can also

see supporting my SSDs, LOL.

Spoiler

 

Last but not least for now, there's FreeNAS 11.2 running (or booting)! Really happy that worked out almost flawlessly.

I've had a little trouble figuring out the way actually assign Volumes as that is quite confusing imo.

Starting of you have to make a Pool, which was easy enough. I just selected my two 860Evos for a striped Raid0.

Then you need a Dataset which is like a Folder on the Raid Array or drive. To access this drive via windows however,

you need to assign a Share to the Folder and assign a User (which you have to creat first) to the Share with a username and password.

Only then (and after some Network configuration which is pretty straight forward though) can you access the NAS in Windows (or Linux or uuhh, Mac, if you are into that....)

 

 

A Funny thing was, that FreeNAS itself chose the IP ending .69 which ofc, i will keep

You always enter the Box over the Web-UI with this IP address so don't lose it  

Spoiler

 

 

 

 

So for now, thanks for reading. If you have any comments or question, feel free to ask and i'll do my best to answer and hopefully someone

else will pic this up to also build himself a FreeNAS. Or you could just wait to see how crazy it'll drive me first 

 

 

 

[LukeSavenije]

you went for bossnas? not a bad choise. I would say this one is bananas

[Spotty]

3 minutes ago, FloRolf said:

here are a few pics of the components i'm using

First off the Drives i already purchased. Very beautiful! Really enjoyed opening these puppies up! :3

You get your HDDs in retail boxes? I've only ever got the anti static bags...

 

4 minutes ago, FloRolf said:

They also have small heat issues when running the system, so does anyone have an idea to keep them

a little cooler? I was thinking about just glueing small vram heatsinks on them.

I probably wouldn't worry about it. I've been running FreeNAS off a cheap USB for a year and haven't had any issues... Yet...

 

8 minutes ago, FloRolf said:

I've had a little trouble figuring out the way actually assign Volumes as that is quite confusing imo.

Starting of you have to make a Pool, which was easy enough. I just selected my two 860Evos for a striped Raid0.

Then you need a Dataset which is like a Folder on the Raid Array or drive. To access this drive via windows however,

you need to assign a Share to the Folder and assign a User (which you have to creat first) to the Share with a username and password.

Only then (and after some Network configuration which is pretty straight forward though) can you access the NAS in Windows (or Linux or uuhh, Mac, if you are into that....) 

Well, that's pretty much it. If you haven't already, assign the network shares a drive letter in windows to make your life easier. (In Windows right click on the share folder and click Map Network Drive).

[Crunchy Dragon]

20 minutes ago, FloRolf said:

They also have small heat issues when running the system, so does anyone have an idea to keep them

a little cooler? I was thinking about just glueing small vram heatsinks on them.

You could get a small fan to blow air over them.

[FloRolf]

1 hour ago, LukeSavenije said:

you went for bossnas? not a bad choise. I would say this one is bananas

Yeah, it's pretty bananas but not enough to come up in my network

 

1 hour ago, Spotty said:

You get your HDDs in retail boxes? I've only ever got the anti static bags...

 

I probably wouldn't worry about it. I've been running FreeNAS off a cheap USB for a year and haven't had any issues... Yet...

 

Well, that's pretty much it. If you haven't already, assign the network shares a drive letter in windows to make your life easier. (In Windows right click on the share folder and click Map Network Drive).

Yeah no, the ones in retail boxes are literally 30€ more expensive. They do however come with 3 instead of 2 years warranty and I got mine in an Amazon sale. The other 4 drives I'll get will probably be bulk in just bags. 

 

1 hour ago, Crunchy Dragon said:

You could get a small fan to blow air over them.

Not sure if I want a fan outside of the case and a tiny one at that. 

I'll just see how they run and then look for a solution later on. 

Right now without any moving HDDs, the system is dead silent. NH-D14 is amazing and so is the PSU. Having no GPU also helps with noise

[FloRolf]

Update time already

 

So with the resources i have right now i was able to achieve roughly 110mb/s read and write speeds from my PC to the NAS, which is pretty much 

what we can expect from a 1gbit connection. I'm pretty happy though as that means for now everything is working just fine.

 

With those limited speeds i non the less tried gaming from NAS to PC.

I Installed Psychonauts on the NAS and measured the time it takes the game to start and the time it takes to load into the actual game.

Both times i got the exact same result of 16 Seconds for each. This means playing (this game) from NAS is exactly as fast as playing it from my external Raid0 HDD array which i'm more than happy with. FPS also didnt take a hit and everything behaved normal! NICE!

 

Here's another update:

Apparently FreeNAS accepts the Dual 10Gbit NIC i impulse ordered a couple months ago. Win7 wasnt too happy about it but i could configure it in FreeNAS. Problem is, 

i can't do any testing yet as it's the only 10Gbit card i have. Welp....

Spoiler

 

[FloRolf]

Aaaaand another update!

 

Went full SYW and got myself a used but perfect condition Fractal Design R5 which even has acustic insulation.

I know it's quite huge for a NAS but due to my need of the ATX board and at least 6 HDD Slots it's the best I could think of. Plus is was just 50€. 

 

Here are a few pics of:

The case:

Spoiler

 

 

The system finally inside the case and not full ghett0 mode on the tablet:

Spoiler

 

 

Also some nicer SSD mounting and shitty cable management. I'll remanage the cables once I have all of my HDDs. 

One issue is, that the SATA power cable supplied by the PSU is not made for SSDs installed next to each other so I had to use a Molex to SATA adapter. i'll probably order another SATA PSU cable from Seasonic though so i dont need Molex anymore.

Spoiler

 

[Nicnac]

Now I feel like copying if I ever wanna build a NAS  Thats name is just too good

[Technomancer__]

hmm 2 usb drives. I just have mine to auto backup on the array.

btw use handbrake to compress your media. I use h.265 at slower with rf set on 22. it takes a day to encode with a ryzen 2700x but the compression ratio is amazing and I cant notice any difference.

[Windows7ge]

Some things should be addressed here to prevent future mis-information.

 

On 12/7/2018 at 10:33 AM, FloRolf said:

1x 250gb Samsung 970Evo as Cache

If you mean cache for the zpool it's not going to work like you think it is. Putting cache or a log on a zfs pool is only good for accelerating synchronous reads/writes. Operations such as hosting VMs or Databases for example. As network storage most if not all operations to the pool are asynchronous, data will only be written to the pool at the speed the pool can accept it and the same goes for reads if RAM isn't be used.

 

To further reason why you don't need a cache drive for a zfs pool. zfs uses excess RAM as a read cache. Frequently accessed files will be cached in RAM and read from RAM when requested. This is why more RAM is generally better for zfs when handling large amounts of data.

 

On 12/7/2018 at 10:33 AM, FloRolf said:

FreeNAS recommends 1gb RAM for each TB of Storage

This is a misconception. One of the developers isn't even sure where this misconception began:

 

 

On 12/7/2018 at 10:33 AM, FloRolf said:

RAM: 16gb G.Skill Aegis 3000mhz

ECC memory is generally more recommended for FreeNAS. If the rare occasion were to occur where a bit flips in RAM FreeNAS won't be able to detect that before writing it to the pool and corrupting data.

 

On 12/7/2018 at 10:33 AM, FloRolf said:

5. VMs
-> Actually i've never had any interest in VMs but i wanna give it a try, maybe to learn some better about
Ubuntu or whatever. This is also a low priority but still interesting.

 

For virtualization FreeNAS uses Bhyve. From my experience it is less than optimal. I can't say I've successfully installed Ubuntu or any other Linux OS but that was quite some time ago. I do know Windows should install fine but understand that the frame rate you'll see will be in the magnitude of 15 or less.

 

I've been using FreeNAS for the past 4 years or so. If you have any setup, tweak, or trouble-shoot questions let me know.

[FloRolf]

3 hours ago, Nicnac said:

Now I feel like copying if I ever wanna build a NAS  Thats name is just too good

Name is ™

 

3 hours ago, Technomancer__ said:

hmm 2 usb drives. I just have mine to auto backup on the array.

btw use handbrake to compress your media. I use h.265 at slower with rf set on 22. it takes a day to encode with a ryzen 2700x but the compression ratio is amazing and I cant notice any difference.

Yeah I figured two usb is a safe way to go. Ofc I'll probably also backup on the array aswell. 

Not sure if I'll compress my stuff. I mean, I buy Blurays because of the quality over Netflix and if my network can handle it, why not stream in full quality? 

 

3 hours ago, Windows7ge said:

Some things should be addressed here to prevent future mis-information.

 

If you mean cache for the zpool it's not going to work like you think it is. Putting cache or a log on a zfs pool is only good for accelerating synchronous reads/writes. Operations such as hosting VMs or Databases for example. As network storage most if not all operations to the pool are asynchronous, data will only be written to the pool at the speed the pool can accept it and the same goes for reads if RAM isn't be used.

 

To further reason why you don't need a cache drive for a zfs pool. zfs uses excess RAM as a read cache. Frequently accessed files will be cached in RAM and read from RAM when requested. This is why more RAM is generally better for zfs when handling large amounts of data.

 

This is a misconception. One of the developers isn't even sure where this misconception began:

 

 

ECC memory is generally more recommended for FreeNAS. If the rare occasion were to occur where a bit flips in RAM FreeNAS won't be able to detect that before writing it to the pool and corrupting data.

 

For virtualization FreeNAS uses Bhyve. From my experience it is less than optimal. I can't say I've successfully installed Ubuntu or any other Linux OS but that was quite some time ago. I do know Windows should install fine but understand that the frame rate you'll see will be in the magnitude of 15 or less.

 

I've been using FreeNAS for the past 4 years or so. If you have any setup, tweak, or trouble-shoot questions let me know.

First off, thanks m8, I really appreciate it! Hope to see you around this thread to debunk some mysteries and show my mistakes

 

About the cache, the idea was to cache game data mainly. Ofc streaming a movie or storing a picture won't benefit but the intention was that playing a certain game for a couple weeks will essentially cache it into the SSD and load it up faster. My "small" 16gb ram won't suffice. Correct me if I'm wrong for this kind of situation. If it doesn't work out or I won't see any noticeable difference I'll just remove the drive and put it in another PC as boot drive also, my RaidZ2 should in theory (https://calomel.org/zfs_raid_speed_capacity.html) be pretty fast already so it might not be worth it at all. 

 

Second, yes I know 1gb/1tb isn't actually true but more is always better and I think 16gb isn't overkill by any means. 

As for ECC, i might replace mine with ecc once prices come down. Good thing is, I can always swap stuff out and reuse this ram in other systems. Do you think it's worth it in my use cases though? I'm not doing any serious work, just saving pics and streaming media. 

 

From what I've read VMs work quite fine on FreeNAS but it's just for basic VM experience anyways so if it doesn't work, it doesn't matter

 

Thanks again! 

[Windows7ge]

2 hours ago, FloRolf said:

About the cache, the idea was to cache game data mainly. Ofc streaming a movie or storing a picture won't benefit but the intention was that playing a certain game for a couple weeks will essentially cache it into the SSD and load it up faster.

The only problem here is you'd be calling for data asynchronously (game data in this instance). The SSD will only accelerate synchronous operations (again, like VM's or databases). Assuming the game successfully runs off the server, within the first day or two you'd see your games load faster because the system would cache the game files in excess RAM. If it does use the SSD at all it will copy the files to the SSD but when you want to access them it will only read the data as quickly as it can fetch it from the pool really defeating the purpose of even having it.

 

2 hours ago, FloRolf said:

my RaidZ2 should in theory (https://calomel.org/zfs_raid_speed_capacity.html) be pretty fast already so it might not be worth it at all. 

With a 6 drive pool you should already see the performance of about an SSD when utilizing 10Gbit. How well the latency will be fetching data when you want to play a game I can't determine.

 

2 hours ago, FloRolf said:

Second, yes I know 1gb/1tb isn't actually true but more is always better and I think 16gb isn't overkill by any means. 

As for ECC, i might replace mine with ecc once prices come down. Good thing is, I can always swap stuff out and reuse this ram in other systems. Do you think it's worth it in my use cases though? I'm not doing any serious work, just saving pics and streaming media. 

The more you have the more you can cache. This benefits 10Gbit a lot when using a sub-10Gbit capable pool. It should also be noted that ZFS isn't a performance oriented file system. Its primary objective is to protect/prevent data loss. With enough tweaking you can see decent performance but if you want raw speed there are better file systems. BTRFS comes to mind.

 

In all honesty for your application ECC memory isn't a real necessity. It's only marginally more expensive and it's recommended for protecting the data so it's the standard. Since AMD processors support ECC memory it's a future option if you ever find it necessary to have it. Though I don't know if the motherboard will allow it.

 

2 hours ago, FloRolf said:

From what I've read VMs work quite fine on FreeNAS but it's just for basic VM experience anyways so if it doesn't work, it doesn't matter

Bhyve (the VM software on FreeNAS) does work well enough for general testing/messing around but if you ever wanted to REALLY get into it there are better options.

[TVwazhere]

On 12/7/2018 at 10:33 AM, FloRolf said:

Hello, Guten Tag and Bonjour everyone,  

 

Blah blah blah NAS stuff

On 12/7/2018 at 12:07 PM, FloRolf said:

blah blah blah update

On 12/7/2018 at 3:35 PM, FloRolf said:

Update time already

 

Blah blah blah more updates

*snooze*

23 hours ago, FloRolf said:

got myself a used but perfect condition Fractal Design R5 

You have my attention

Spoiler

/s

 

Nice NAS though!

 

[FloRolf]

On 12/9/2018 at 11:52 PM, Windows7ge said:

The only problem here is you'd be calling for data asynchronously (game data in this instance). The SSD will only accelerate synchronous operations (again, like VM's or databases). Assuming the game successfully runs off the server, within the first day or two you'd see your games load faster because the system would cache the game files in excess RAM. If it does use the SSD at all it will copy the files to the SSD but when you want to access them it will only read the data as quickly as it can fetch it from the pool really defeating the purpose of even having it.

 

With a 6 drive pool you should already see the performance of about an SSD when utilizing 10Gbit. How well the latency will be fetching data when you want to play a game I can't determine.

 

The more you have the more you can cache. This benefits 10Gbit a lot when using a sub-10Gbit capable pool. It should also be noted that ZFS isn't a performance oriented file system. Its primary objective is to protect/prevent data loss. With enough tweaking you can see decent performance but if you want raw speed there are better file systems. BTRFS comes to mind.

 

In all honesty for your application ECC memory isn't a real necessity. It's only marginally more expensive and it's recommended for protecting the data so it's the standard. Since AMD processors support ECC memory it's a future option if you ever find it necessary to have it. Though I don't know if the motherboard will allow it.

 

Bhyve (the VM software on FreeNAS) does work well enough for general testing/messing around but if you ever wanted to REALLY get into it there are better options.

Are you sure about the game being asynchronous? First time sure, but after having played it a couple times? 

Well whatever, we'll see how it goes. 

 

Yeah the RaidZ2 should read/write about 450mb/s, so almost ssd speeds. Should be more than ideal for anything really. The Raid0 ssd array however should fully utilize the 10gbit link. 

 

About the VMs again, I'm using 11.2 already which has a dedicated VM Tab, so maybe they improved support. I'll definitely post my experience once I've tried it though so keep an eye out! 

 

On 12/10/2018 at 3:25 PM, TVwazhere said:

*snooze*

You have my attention

  Reveal hidden contents

/s

 

Nice NAS though!

 

Honestly now, this case is amaze-balls! With the Noctua cooler and the Seasonic PSU I can hear a single thing. I was afraid at the beginning that the HDD cages were just bare metal but it came with a bunch of small rubber parts so it should isolate the drives pretty well. 

 

 

UPDATE TIME WHOOOO:

 

This just arrived today shipping from Bulgaria off of eBay. 

It's not one, but two.... 

 

Spoiler

 

Dual 10gbit NICs by Chelsio! Whoooiiiio! 

Additionally I got a single 3m SFP+ DAC cable. 

Can't wait to try them out but I have to wait until the weekend, maybe even Sunday ;(

 

 

 

 

So stay tuned peeps

 

[Windows7ge]

56 minutes ago, FloRolf said:

Are you sure about the game being asynchronous? First time sure, but after having played it a couple times? 

Well whatever, we'll see how it goes. 

I should be a bit more specific when I said asynchronous. This is a server-side operation. It works on a layer lower than the network layer. In order for you to store files, games, music, videos, etc on the server you setup SMB/CIFS on the server (windows). This protocol when set to store data to a zpool writes EVERYTHING asynchronously. It does not matter what you are sending or pulling to/from the server if you're doing it over the SMB service it's being done asynchronously. Now let's say you setup a jail and installed MySQL which is a fairly well known database software. This writes data to the database synchronously so when written to the pool it'd cache successfully in the SSD and when needed it can read from the SSD. Unfortunately you'd require a mass number of clients to make this worth doing. In your case you will see a little improvement after a couple uses because files will automatically cache in RAM on their own. The SSD will just kind of be there not getting used or files will copy to it but not get read.

 

1 hour ago, FloRolf said:

Yeah the RaidZ2 should read/write about 450mb/s, so almost ssd speeds. Should be more than ideal for anything really. The Raid0 ssd array however should fully utilize the 10gbit link. 

This is true. What is it you're trying to achieve with the 2 SSD RAID0? High speed is only good for large files and 1TB can fill up fast. If it's temporary storage for things like video editing then it's not a bad idea.

 

1 hour ago, FloRolf said:

About the VMs again, I'm using 11.2 already which has a dedicated VM Tab, so maybe they improved support. I'll definitely post my experience once I've tried it though so keep an eye out! 

I'm on FreeNAS-11.1-U6 it also has a dedicated VMs tab. You can check the Bhyve version by running the following command from console:

vm version

This is my output:

vm-bhyve: Bhyve virtual machine management v1.1-p8 (build 101073)

You can experiment with the VMs tab. I did, there's a few useful things a simple VM can do for you in the background assuming the host OS isn't compromised.

[FloRolf]

On 12/13/2018 at 10:19 PM, Windows7ge said:

I should be a bit more specific when I said asynchronous. This is a server-side operation. It works on a layer lower than the network layer. In order for you to store files, games, music, videos, etc on the server you setup SMB/CIFS on the server (windows). This protocol when set to store data to a zpool writes EVERYTHING asynchronously. It does not matter what you are sending or pulling to/from the server if you're doing it over the SMB service it's being done asynchronously. Now let's say you setup a jail and installed MySQL which is a fairly well known database software. This writes data to the database synchronously so when written to the pool it'd cache successfully in the SSD and when needed it can read from the SSD. Unfortunately you'd require a mass number of clients to make this worth doing. In your case you will see a little improvement after a couple uses because files will automatically cache in RAM on their own. The SSD will just kind of be there not getting used or files will copy to it but not get read.

 

This is true. What is it you're trying to achieve with the 2 SSD RAID0? High speed is only good for large files and 1TB can fill up fast. If it's temporary storage for things like video editing then it's not a bad idea.

 

I'm on FreeNAS-11.1-U6 it also has a dedicated VMs tab. You can check the Bhyve version by running the following command from console:

vm version

This is my output:

vm-bhyve: Bhyve virtual machine management v1.1-p8 (build 101073)

You can experiment with the VMs tab. I did, there's a few useful things a simple VM can do for you in the background assuming the host OS isn't compromised.

 

So yeah, i have the same Bhyve Version as you do so it might also suck

And also yeah, SSD Raid array is supposed to be for temporary files and current projects.

 

 

 

You might be able to help me in the

newest Boss-NAS Update

 

I just installed the second 10Gbit NIC in my main PC after finding out the Crucial MX SSD in my GF's system f#cking sucks! That thing is so damn slow it's painfull.

Running Crystal Disk Mark on it only resulted in roughly 200-300MB/s speeds which is terrible and definetly not enough to saturate a 10Gbit link.

This is where my Main Rig with at least a 840Evo comes into play.

 

Spoiler

 

Because who needs a dGPU anyways, right? Running on Intel HD3000

 

 

But for some reason after having the card installed, i ran into a couple BSODs. Those however were resolved (???) after 3 reboots. No idea what happened here.

Spoiler

 

 

Now here comes the funny part:

What the actual fuck? lol. While this is seriously hilarious, what is going on? On some benchmarks it even goes down to 25MB/s

The write speed is totally legit for 2x860s in Raid0 but the read? After copying files from left to right i found that this is the actual speed.

 

Reaching the 10Gbit during file transfer to the NAS is quite hard however.

Tried copying Steam games topped out at 250MB mostly with spikes to 800MB (impossible on my SSD actually?) at the beginning of the transfer.

 

Even more funny is this, running over 1Gbit connection. This is obviously also correct but why are read speeds double than on 10Gbit?

 

 

Is there something wrong with the NIC?

i'll try it's second port now to see if that helps but i don't think so as write is already amazing. 

 

 

 

I've got so say though, when 10Gbit works, it's insane. I copied a 3GB Video to the NAS which gave me the full 10gbit and it was literally 3 Seconds. Nice.

[Windows7ge]

27 minutes ago, FloRolf said:

i ran into a couple BSODs

"A driver tried to access an address that is pageable (or that is completely invalid) while the IRQL was too high.

This bug check is usually caused by drivers that have used improper addresses." - docs.microsoft.com

 

If the system figured out the address range after a few reboots you should be fine now.

 

32 minutes ago, FloRolf said:

What the actual fuck? lol. While this is seriously hilarious, what is going on? On some benchmarks it even goes down to 25MB/s

The write speed is totally legit for 2x860s in Raid0 but the read? After copying files from left to right i found that this is the actual speed.

 

Reaching the 10Gbit during file transfer to the NAS is quite hard however.

Tried copying Steam games topped out at 250MB mostly with spikes to 800MB (impossible on my SSD actually?) at the beginning of the transfer.

I had a feeling you might hit this. I decided not to say anything though. There are some tweaks you can do but it has to be understood that ZFS isn't the fastest file system. Though something is clearly wrong here. It also has to be understood that SMB support on linux is SAMBA's responsibility. Although it works and works fairly reliably the efficiency isn't there and so performance tends to suffer.

 

If I recall you already toyed with jumbo packets.

What you can also toy with are advanced options and tunables.

 

If you go to Services -> SMB -> Auxiliary Parameters:

You can cut & paste the following:

server multi channel support = Yes
strict allocate = No
read raw = Yes
write raw = Yes
server signing = No
strict locking = No
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
min receivefile size = 16384
use sendfile = Yes
aio read size = 16384
aio write size = 16384

The may hurt the write performance but may also help the read. If it does nothing you can delete it all.

 

You can also experiment with tunables however I urge you to back up your configuration and pool keys if they're encrypted (if you don't know how, ask) in the event you mis-type a tunable as it can brick FreeNAS.

 

Go to System -> Tunables and type in the following list:

 

Using pure SSD storage is a bit complicated on ZFS (trust me some people here can vouch for me that I've done it) it doesn't respond the way you think it would and requires a lot of toying around to get anywhere close to what you wanted.

 

I have one other thing you can try that would just about guarantee 1GB/s read/write but I don't recommend it at all. I'll mention it to you if we reach that road.

[FloRolf]

3 hours ago, Windows7ge said:

"A driver tried to access an address that is pageable (or that is completely invalid) while the IRQL was too high.

This bug check is usually caused by drivers that have used improper addresses." - docs.microsoft.com

 

If the system figured out the address range after a few reboots you should be fine now.

 

I had a feeling you might hit this. I decided not to say anything though. There are some tweaks you can do but it has to be understood that ZFS isn't the fastest file system. Though something is clearly wrong here. It also has to be understood that SMB support on linux is SAMBA's responsibility. Although it works and works fairly reliably the efficiency isn't there and so performance tends to suffer.

 

If I recall you already toyed with jumbo packets.

What you can also toy with are advanced options and tunables.

 

If you go to Services -> SMB -> Auxiliary Parameters:

You can cut & paste the following:

server multi channel support = Yes
strict allocate = No
read raw = Yes
write raw = Yes
server signing = No
strict locking = No
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
min receivefile size = 16384
use sendfile = Yes
aio read size = 16384
aio write size = 16384

The may hurt the write performance but may also help the read. If it does nothing you can delete it all.

 

You can also experiment with tunables however I urge you to back up your configuration and pool keys if they're encrypted (if you don't know how, ask) in the event you mis-type a tunable as it can brick FreeNAS.

 

Go to System -> Tunables and type in the following list:

 

Using pure SSD storage is a bit complicated on ZFS (trust me some people here can vouch for me that I've done it) it doesn't respond the way you think it would and requires a lot of toying around to get anywhere close to what you wanted.

 

I have one other thing you can try that would just about guarantee 1GB/s read/write but I don't recommend it at all. I'll mention it to you if we reach that road.

Yeah well it BSOD'ed a couple more times later when I tried switching the ports and all. Hope it'll be stable once it works properly. 

I then encountered another issue where the NIC in the NAS (or PC?) started acting up but I overcame that. I think it was an overheating issue as these Chelsio's really do get hot. Might need a fan for them or a new heatsink. 

 

 

To your suggestions

No, I haven't played with Jumbos yet as the setting doesn't appear in windows where it should be? Do you know if there's another way to set jumbo on the nic? It's quite weird since it was neither on Win10 nor on 7.

This is where I also want to add that on win7 with the shitty ssd I was actually able to reach like 300mb read and running crystaldiskmark even showed a theoretical 700mb read and the 1000 write I'm also seeing on my Win10 machine. 

 

I'll look through your suggestions tomorrow to see if they take any effect. Thanks for now! 

 

Oh and also, even with ZFS not being the fastest, I don't want to achieve the full 1gbit as the RaidZ2 HDDs will cap at ~500mb anyways but ofc I want to try my best to go as high as possible

If it turns out to be stable 500 it'll be fine for me. 

[Windows7ge]

10 hours ago, FloRolf said:

I think it was an overheating issue as these Chelsio's really do get hot. Might need a fan for them or a new heatsink.

Most 10Gbit NICs are designed to be installed in a rack mount server where they get mega heaps of airflow. I mounted a tiny 40mm fan to my desktops SFP+ NIC. Keeps it nice and cool.

 

10 hours ago, FloRolf said:

No, I haven't played with Jumbos yet as the setting doesn't appear in windows where it should be? Do you know if there's another way to set jumbo on the nic? It's quite weird since it was neither on Win10 nor on 7.

Are you looking inside the NIC adapters advanced menu? Changing the MTU is a pretty buried setting. Note you also have to change it on the server side. If you're using a 10Gbit switch it has to have Jumbo Frames enabled as well.

 

10 hours ago, FloRolf said:

This is where I also want to add that on win7 with the shitty ssd I was actually able to reach like 300mb read and running crystaldiskmark even showed a theoretical 700mb read and the 1000 write I'm also seeing on my Win10 machine. 

Crystal disk mark is nice but your workflow may not show you the same numbers. One thing you can try doing is temporarily making a RAMDisk. This would just make sure you're getting what you paid for. But yes, until (if you ever) make the jump to PCI_e based storage the max you'll see will be ~550MB/s-ish real world performance.

[FloRolf]

On 12/17/2018 at 7:23 AM, Windows7ge said:

Most 10Gbit NICs are designed to be installed in a rack mount server where they get mega heaps of airflow. I mounted a tiny 40mm fan to my desktops SFP+ NIC. Keeps it nice and cool.

 

Are you looking inside the NIC adapters advanced menu? Changing the MTU is a pretty buried setting. Note you also have to change it on the server side. If you're using a 10Gbit switch it has to have Jumbo Frames enabled as well.

 

Crystal disk mark is nice but your workflow may not show you the same numbers. One thing you can try doing is temporarily making a RAMDisk. This would just make sure you're getting what you paid for. But yes, until (if you ever) make the jump to PCI_e based storage the max you'll see will be ~550MB/s-ish real world performance.

okay, long time no update.

 

i think i've nailed the problem down to the driver. 

I've been getting tons and tons of BSODs over the last few days and the write speeds didn't go up no matter what.

At this point it doesn't even seem to work at all anymore.

Checking on Chelsios website it says the N320E Driver is only compatible up to Win7. This explains why it works flawlessly on my Win7 PC but not on the Win10.

That's a huge shame as the seller mentions Win10 support on his site (it's eBay though soooo...)

 

I've reran crystaldiscmark on win7 and it was somehow a lot faster than before. while 2 weeks ago it was 700/1000 its now pretty much 1000/1000, lol.

so in theory it does work perfectly  

[Windows7ge]

6 hours ago, FloRolf said:

okay, long time no update.

 

i think i've nailed the problem down to the driver. 

I've been getting tons and tons of BSODs over the last few days and the write speeds didn't go up no matter what.

At this point it doesn't even seem to work at all anymore.

Checking on Chelsios website it says the N320E Driver is only compatible up to Win7. This explains why it works flawlessly on my Win7 PC but not on the Win10.

That's a huge shame as the seller mentions Win10 support on his site (it's eBay though soooo...)

 

I've reran crystaldiscmark on win7 and it was somehow a lot faster than before. while 2 weeks ago it was 700/1000 its now pretty much 1000/1000, lol.

so in theory it does work perfectly  

Yeah I didn't even try sourcing a driver for my 10Gbit NIC. It's a generic card with no name. It just goes by the chipset it uses called the Broadcom BCM57810S thinking about it makes me want to look around again but I'm sure I'll only find drivers for Windows Server. That's if I'm lucky.

 

It's good to hear that things have leveled out performance wise. Hopefully it stays that way. From my experience FreeNAS is really reliable and consistent in its performance...so long as you don't poke too many things in the WebUI or CLI.

 

I have,

on many occasions,

bricked FreeNAS,

because I poked things,

on many occasions.

[mr moose]

2 minutes ago, Windows7ge said:

Yeah I didn't even try sourcing a driver for my 10Gbit NIC. It's a generic card with no name. It just goes by the chipset it uses called the Broadcom BCM57810S thinking about it makes me want to look around again but I'm sure I'll only find drivers for Windows Server. That's if I'm lucky.

 

It's good to hear that things have leveled out performance wise. Hopefully it stays that way. From my experience FreeNAS is really reliable and consistent in its performance...so long as you don't poke too many things in the WebUI or CLI.

 

I have,

on many occasions,

bricked FreeNAS,

because I poked things,

on many occasions.

It's good to hear that it is reliable, I am currently using freenas, and while I haven't got critical things stored on it, I do have things I can't effectively replace.  

 

 

Anyway:

More pics please

[Windows7ge]

21 minutes ago, mr moose said:

It's good to hear that it is reliable, I am currently using freenas, and while I haven't got critical things stored on it, I do have things I can't effectively replace.

That's the + & - of the file system it uses. ZFS (Zetabyte File System) as I've mentioned earlier prioritises data integrity over performance. That's the trade-off.

 

There's also the perk that it doesn't care what SATA controllers you're using. You could mix four different controllers if you wanted, pch ports, marvel port, maybe a PCIe card or two. ZFS doesn't care it'll RAID em.

[mr moose]

25 minutes ago, Windows7ge said:

That's the + & - of the file system it uses. ZFS (Zetabyte File System) as I've mentioned earlier prioritises data integrity over performance. That's the trade-off.

 

There's also the perk that it doesn't care what SATA controllers you're using. You could mix four different controllers if you wanted, pch ports, marvel port, maybe a PCIe card or two. ZFS doesn't care it'll RAID em.

I'm following this thread, because 1 I like the sound of FloRolfs system and 2 I want to upgrade my freenas to a full raid backup.  As of now I have no idea and when it comes to software I am a bit of an old dog (its hard to teach me new tricks).   By the time I am ready it sounds like this thread will contain all the information I need (minus the 10Gb stuff, I'm not going that far).

 

 

[Windows7ge]

8 hours ago, mr moose said:

I want to upgrade my freenas to a full raid backup.  As of now I have no idea and when it comes to software I am a bit of an old dog (its hard to teach me new tricks).   By the time I am ready it sounds like this thread will contain all the information I need

Do you mean backing up data from your computer(s) to your NAS?

Do you mean backing up the server pool data to another pool?

Do you mean backing up the server pool data to a separate server?

 

There's ways to automate all three but the configurations vary. Unless FloRolf wants to set this up too then it's not going to get discussed any time soon. I can help you with that separately though. I've used the first 2 methods and I'm in the process of setting up the third.