Jump to content

Questions About Server Hardware and ZFS

Hello,

 

I am currently in the process of building a new NAS, as I have outgrown my old one.

Currently, I have finished running bad-blocks on my new drives (and discovered that one drive is already having problems) and have decided to use FreeNas this time around.

Previously I had been using Ubuntu server, and it worked well for everything I needed, but the nice web gui in FreeNas is extremely tempting.

 

- I currently have my drives in a desktop system with a core i5 4460 and 16Gb of RAM, I am tempted to leave it in this setup because it is so quiet. I really do like the form factor of rack-mount gear though, and already own a rack, so I'm looking at a 1U or 2U system to replace my PowerEdge 2950. 

 

Here are my questions/concerns-

 

- Can someone explain the difference in running ZFS with and without ECC RAM? I understand there are benefits, but what are they?

- What is a good 2U or 1U system capable of holding 6 drives or more, (that doesn't sound like a jet engine like my 2950) and costs under $500

- Other than badblocking new HDDs as I get them, are there any other tests I should be running before installation to avoid data corruption?

- Can FreeNas easily be set up for link aggregation?

- What are your experiences with virtual machine performance in FreeNas?

 

Thanks in advance for all the help!

Link to comment
Share on other sites

Link to post
Share on other sites

30 minutes ago, Kered124 said:

Can someone explain the difference in running ZFS with and without ECC RAM? I understand there are benefits, but what are they?

From a features and usage perspective, nothing. It works the same. ZFS has no idea your using ECC. It reduces chance of a memory error causing data loss, and ZFS(along with every other filesystem) assumes data in ram is good and doesn't checksum it(but this may be comming in ZFS, but ecc alraedy does it)

 

31 minutes ago, Kered124 said:

- What is a good 2U or 1U system capable of holding 6 drives or more, (that doesn't sound like a jet engine like my 2950) and costs under $500

r710, r510 should both fit this

 

The hp di180g6 and the c2100 are louder, but can be modded easily to be quieter.

 

32 minutes ago, Kered124 said:

- Other than badblocking new HDDs as I get them, are there any other tests I should be running before installation to avoid data corruption?

Check the smart data of the drives and run scrubs.

 

32 minutes ago, Kered124 said:

- Can FreeNas easily be set up for link aggregation?

Yep, but it won't make most transfers faster. For faster nas access id look into 10gbe.

 

33 minutes ago, Kered124 said:

- What are your experiences with virtual machine performance in FreeNas?

Performance is fine, but features is a good amount worse than things like KVM, esxi, hyper-v and xen.

 

If you want ZFS, web gui, and good vm support look at proxmox.

Link to comment
Share on other sites

Link to post
Share on other sites

I had been heavily looking at proxmox, but I really like all of the plugins that come with FreeNas, like Plex. I don't need a ton of features for my virtual machines, just the ability to run some basic tasks and play around a little. I do have an enterasys g3g170-24 switch that is capible of 10gb networking with a module.. Tempting. 

 

I was also looking at using an older ssd I have laying around as a cache, is this easily do-able with FreeNas?

Link to comment
Share on other sites

Link to post
Share on other sites

So there is no direct benefit for running ECC RAM in conjunction with FreeNas other than just letting the ECC RAM do its normal thing? I thought I had heard that FreeNas was capable of leveraging the ECC RAM in some way to be more effective with ZFS. I must have misheard/misread.  

1 hour ago, Electronics Wizardy said:

r710, r510 should both fit this

 

The hp di180g6 and the c2100 are louder, but can be modded easily to be quieter.

Out of the servers mentioned, which do you like best? I have had good luck with my dell servers, but I have never been overly happy with the level of noise they produce compared to more modern servers. I am looking for something much quieter. 

Link to comment
Share on other sites

Link to post
Share on other sites

54 minutes ago, Kered124 said:

Out of the servers mentioned, which do you like best? I have had good luck with my dell servers, but I have never been overly happy with the level of noise they produce compared to more modern servers. I am looking for something much quieter. 

If you want a more quiet server, don't look for a rack server but a tower server instead. Or at least a newer-ish 2U server.

 

That said, the R710 is pretty okay noise wise and so are HPE's DL380 rack servers. They're not commonly found with 3,5" drive bays, though. Most 1U and 2U rackmount servers nowadays are 2,5" only so that's something to keep in mind. The units with 3,5" bays often commend a premium and a 2U server can usually only hold 4-8 disks compared to 8-16 2,5" disks.

PC Specs - AMD Ryzen 7 5800X3D MSI B550M Mortar - 32GB Corsair Vengeance RGB DDR4-3600 @ CL16 - ASRock RX7800XT 660p 1TBGB & Crucial P5 1TB Fractal Define Mini C CM V750v2 - Windows 11 Pro

 

Link to comment
Share on other sites

Link to post
Share on other sites

8 hours ago, Kered124 said:

I had been heavily looking at proxmox, but I really like all of the plugins that come with FreeNas, like Plex. I don't need a ton of features for my virtual machines, just the ability to run some basic tasks and play around a little. I do have an enterasys g3g170-24 switch that is capible of 10gb networking with a module.. Tempting. 

 

I was also looking at using an older ssd I have laying around as a cache, is this easily do-able with FreeNas?

The vm system in freenas leaves a lot to be desired, proxmox is just much nicer.

 

 

You can easily add a ssd cache or l2-arc, but it often doesn't help that much for most uses.

 

 

7 hours ago, Kered124 said:

So there is no direct benefit for running ECC RAM in conjunction with FreeNas other than just letting the ECC RAM do its normal thing? I thought I had heard that FreeNas was capable of leveraging the ECC RAM in some way to be more effective with ZFS. I must have misheard/misread.  

Out of the servers mentioned, which do you like best? I have had good luck with my dell servers, but I have never been overly happy with the level of noise they produce compared to more modern servers. I am looking for something much quieter. 

Noise wise the 710 is a good amount better than the 2950's.

 

the r510 might be a good option here due to the large amount of drive bays. If noise is a issue you can change the fan speed in the idrac or mod the fans.

Link to comment
Share on other sites

Link to post
Share on other sites

9 hours ago, Kered124 said:

like Plex.

Plex is stupid IMO

Make me pay to watch my movies on my hardware (e.g. phone) that are stored on my server. (also tried to force me to create an online account to login to my internal server)

 

I use Emby server

Can Anybody Link A Virtual Machine while I go download some RAM?

 

Link to comment
Share on other sites

Link to post
Share on other sites

FreeNAS plugins are a drying breed given there's a "big" change coming up. They're pushing for behyve + docker for things like plex. You could instead create a jail and install the FreeBSD package for plex, but that comes with complications as well. I would use a windows/linux computer, and just mount a share and point plex to it.

 

If you can find a rack mount case cheap enough, you could just transfer the guts of your existing computer into it. However an R710 or R510 should only run you around $300, and some of these cases cost about that, so shoot for <$100 if possible.

Link to comment
Share on other sites

Link to post
Share on other sites

22 hours ago, unijab said:

Plex is stupid IMO

Make me pay to watch my movies on my hardware (e.g. phone) that are stored on my server. (also tried to force me to create an online account to login to my internal server)

 

I use Emby server

<3 Emby. Their app is paid too, but at least the browser is free :)  I run it on my Synology NAS and it's quite nice. My previous NAS had Plex and my current one doesn't support it so I tried Emby and I'm sold.

PC Specs - AMD Ryzen 7 5800X3D MSI B550M Mortar - 32GB Corsair Vengeance RGB DDR4-3600 @ CL16 - ASRock RX7800XT 660p 1TBGB & Crucial P5 1TB Fractal Define Mini C CM V750v2 - Windows 11 Pro

 

Link to comment
Share on other sites

Link to post
Share on other sites

On 7/1/2018 at 8:40 PM, Kered124 said:

- Can someone explain the difference in running ZFS with and without ECC RAM? I understand there are benefits, but what are they?

- What is a good 2U or 1U system capable of holding 6 drives or more, (that doesn't sound like a jet engine like my 2950) and costs under $500

- Other than badblocking new HDDs as I get them, are there any other tests I should be running before installation to avoid data corruption?

- Can FreeNas easily be set up for link aggregation?

- What are your experiences with virtual machine performance in FreeNas?

 

Thanks in advance for all the help!

- There is nothing special about ZFS that requires the use of ECC more so than any other file system. A bad block in ram will mean a bad write. That being said, you should always use ECC in enterprise. For home use it's up to you.

 

- Every u1 or u2 will sound like a jet. You might look into pedestal servers for your house.

 

- Your on ZFS, you don't need to worry, if the data block is bad it will repair it silently with data from the array mirror. ZFS won't return a bad block to the OS. So it can't give you corrupted data. This was a design implementation of it so you can always verify the data your application receives is exactly what it wrote.

 

- FreeBSD can.. not sure if FreeNAS has an option in the Web UI to do that, but it should be possible to implement even if it doesn't. It's literally a one liner sometimes.  "ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1 10.0.0.15/24"

https://www.freebsd.org/doc/handbook/network-aggregation.html

 

- You mean Bhyve? Bhyve is pretty good actually for how new it is. The one huge advantage it has over others is it's use of ZFS and ZVOLS. It makes the virtual disk war going on with other OS's look like clown calling. ("My Cow2 is better than your Cow" lol) It also has Jails that are a thinner virtual layer than any hypervisor.

"Only proprietary software vendors want proprietary software." - Dexter's Law

Link to comment
Share on other sites

Link to post
Share on other sites

Alright, I have a disk that is used that performed poorly on the a badblocks run- producing 16 read errors. Is the disk unusable at this point? No other errors were produced.

 

I now understand that there is no direct benefit to the ZFS file-system from using ECC RAM.

I am currently looking at a dell poweredge r510 for an upgrade, as it seems to be a more quiet system.

Since I already own the Plex app for android, and a roku as well as a few other devices that use plex, I would sort of like to keep using it unless there are other solutions that tie in nicely with the ruku and mobile devices.

 

Thanks for all of the input so far!

Link to comment
Share on other sites

Link to post
Share on other sites

On 7/4/2018 at 10:22 AM, jde3 said:

* snip*

- FreeBSD can.. not sure if FreeNAS has an option in the Web UI to do that, but it should be possible to implement even if it doesn't. It's literally a one liner sometimes.  "ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1 10.0.0.15/24"

https://www.freebsd.org/doc/handbook/network-aggregation.html

*snip*

You can setup link aggregation in the web UI for FreeNAS

 

LA.PNG.3a3afe869737cd49f1e9fa8bbee41153.PNG

There's no place like ~

Spoiler

Problems and solutions:

 

FreeNAS

Spoiler

Dell Server 11th gen

Spoiler

 

 

 

 

ESXI

Spoiler

 

 

 

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

On 7/3/2018 at 10:33 PM, Mikensan said:

FreeNAS plugins are a drying breed given there's a "big" change coming up. They're pushing for behyve + docker for things like plex. You could instead create a jail and install the FreeBSD package for plex, but that comes with complications as well. I would use a windows/linux computer, and just mount a share and point plex to it.

 

If you can find a rack mount case cheap enough, you could just transfer the guts of your existing computer into it. However an R710 or R510 should only run you around $300, and some of these cases cost about that, so shoot for <$100 if possible.

Plugins are switching to iocage jails and will be much simpler once they release 11.2 which is coming in a few months.  Bhyve and docker are already there but you will still get the best performance running jails.  What issues have you had using Plex in a jail?  It's super simple to setup as long as you have the correct permissions.  

Link to comment
Share on other sites

Link to post
Share on other sites

On 7/8/2018 at 3:09 AM, Planet86 said:

Plugins are switching to iocage jails and will be much simpler once they release 11.2 which is coming in a few months.  Bhyve and docker are already there but you will still get the best performance running jails.  What issues have you had using Plex in a jail?  It's super simple to setup as long as you have the correct permissions.  

No issues setting up jails, but you have to be mindful to mount persistent storage. As the base kernel for freenas updates the jail does not, and in order to update the jail you have to rebuild it - without persistent storage you'll loose everything. Which one might think "fine I'll leave it at a slightly older version of BSD" until the port requires a kernel update :-(. 

 

**Also what will the move to iocage mean for users using warden?

Link to comment
Share on other sites

Link to post
Share on other sites

Alright, small update.

 

I purchased 2 brand new (really cheap) hard drives, seagate barracudas. One of them threw 4 read errors in the badblocks run... Should I be concerned?

Link to comment
Share on other sites

Link to post
Share on other sites

On 7/2/2018 at 2:59 AM, Kered124 said:

So there is no direct benefit for running ECC RAM in conjunction with FreeNas other than just letting the ECC RAM do its normal thing? I thought I had heard that FreeNas was capable of leveraging the ECC RAM in some way to be more effective with ZFS. I must have misheard/misread.  

Out of the servers mentioned, which do you like best? I have had good luck with my dell servers, but I have never been overly happy with the level of noise they produce compared to more modern servers. I am looking for something much quieter. 

Here's a doc that explains it:

http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

 

Essentially, there's a tiny tiny chance of scrub failure, in which during a scrub, it might overwrite good data with corrupt data.

 

Basically, it's not all that common, and this can happen anyway with non-ZFS systems.

On 7/2/2018 at 11:56 AM, unijab said:

Plex is stupid IMO

Make me pay to watch my movies on my hardware (e.g. phone) that are stored on my server. (also tried to force me to create an online account to login to my internal server)

 

I use Emby server

Huh? There's a perfectly good working free version of Plex.

 

You can pay for Plex Pass that enables some advanced features, such as simplified remote-access to your media, but I've used Plex for literal years perfectly fine without paying. I only recently got a life-time Plex Pass because I value the product and wanted to contribute.

 

Furthermore, you're not paying to watch your movies on your hardware. You're paying for a beautiful and well made software. Don't want to pay? Use the free version, or use an alternative.

 

You also don't need an account for a local server, but you will need to adjust settings to disable local authentication.

For Sale: Meraki Bundle

 

iPhone Xr 128 GB Product Red - HP Spectre x360 13" (i5 - 8 GB RAM - 256 GB SSD) - HP ZBook 15v G5 15" (i7-8850H - 16 GB RAM - 512 GB SSD - NVIDIA Quadro P600)

 

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, Kered124 said:

Alright, small update.

 

I purchased 2 brand new (really cheap) hard drives, seagate barracudas. One of them threw 4 read errors in the badblocks run... Should I be concerned?

RMA the one with read errors (or return and exchange at the retailer).

 

You want the drives to pass badblocks perfectly with no errors. No point risking your data if it's giving errors from day one.

For Sale: Meraki Bundle

 

iPhone Xr 128 GB Product Red - HP Spectre x360 13" (i5 - 8 GB RAM - 256 GB SSD) - HP ZBook 15v G5 15" (i7-8850H - 16 GB RAM - 512 GB SSD - NVIDIA Quadro P600)

 

Link to comment
Share on other sites

Link to post
Share on other sites

On 7/7/2018 at 1:30 PM, Kered124 said:

I now understand that there is no direct benefit to the ZFS file-system from using ECC RAM.

I am currently looking at a dell poweredge r510 for an upgrade, as it seems to be a more quiet system.

Since I already own the Plex app for android, and a roku as well as a few other devices that use plex, I would sort of like to keep using it unless there are other solutions that tie in nicely with the ruku and mobile devices.

 

Thanks for all of the input so far!

Emby.

It's Roku app is really good.

 

I prefer it over plex.. they are similar but.. it just seems better.

"Only proprietary software vendors want proprietary software." - Dexter's Law

Link to comment
Share on other sites

Link to post
Share on other sites

Alright, ill contact amazon for a return. The past 2 seagate drives I ordered from them have gad bad blocks from day 1, I don't understand the problem. I have never had great luck with seagate drives as far as lifespan is concerned, but I'm surprised to see any brand new drives with bad blocks.

 

Link to comment
Share on other sites

Link to post
Share on other sites

5 minutes ago, Kered124 said:

Alright, ill contact amazon for a return. The past 2 seagate drives I ordered from them have gad bad blocks from day 1, I don't understand the problem. I have never had great luck with seagate drives as far as lifespan is concerned, but I'm surprised to see any brand new drives with bad blocks.

 

Any brand new HDD can arrive with bad sectors on them. It's nothing to do with Seagate themselves or how reliable the drives are.

 

That can happen on any drive from any manufacturer.

 

With that in mind? WDC makes great drives too, so maybe check one of them out.

For Sale: Meraki Bundle

 

iPhone Xr 128 GB Product Red - HP Spectre x360 13" (i5 - 8 GB RAM - 256 GB SSD) - HP ZBook 15v G5 15" (i7-8850H - 16 GB RAM - 512 GB SSD - NVIDIA Quadro P600)

 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, dalekphalm said:
23 minutes ago, Kered124 said:

 

Any brand new HDD can arrive with bad sectors on them. It's nothing to do with Seagate themselves or how reliable the drives are.

 

That can happen on any drive from any manufacturer.

 

With that in mind? WDC makes great drives too, so maybe check one of them out.

I absolutely understand that it can happen with any manufacturer, and I don't hold this against seagate. I am more-so wondering if it is from shipping, as I noticed the packaging wasn't the best. I have a few of their NAS drives and love them, they have been extremely reliable and perform well for my use case. 

 

I thought about trying some of the wdc blue drives or green drives, but I simply bought the cheapest drives available for the size with the full intent of just replacing them as they die.

 

Have you experienced any performance issues with plex? When doing some testing, I noticed that streaming to another computer with the plex player, everything is fine, but when streaming to the roku ultra, there is buffering every 30 seconds or so on a wired connection and 1080p content.

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Kered124 said:

I absolutely understand that it can happen with any manufacturer, and I don't hold this against seagate. I am more-so wondering if it is from shipping, as I noticed the packaging wasn't the best. I have a few of their NAS drives and love them, they have been extremely reliable and perform well for my use case. 

Shipping can definitely kill drives. For a long time, NewEgg had a reputation as the HDD killer, because they would insufficiently pack individual drives for shipment. These days they do a better job though.

1 minute ago, Kered124 said:

I thought about trying some of the wdc blue drives or green drives, but I simply bought the cheapest drives available for the size with the full intent of just replacing them as they die.

That's typically what I do too. I'll research the cheapest drive in any given class (Eg: when I bought the drives for my FreeNAS server, I ended up getting a bunch of Toshiba 3TB drives, because they were cheaper than the WD Blue/Green/Seagate, etc).

1 minute ago, Kered124 said:

Have you experienced any performance issues with plex? When doing some testing, I noticed that streaming to another computer with the plex player, everything is fine, but when streaming to the roku ultra, there is buffering every 30 seconds or so on a wired connection and 1080p content.

I have no performance issues with mine. I have a Roku 4K Premiere and I don't get any buffering unless it's too high bitrate for my WIFI - in such a case, I just go in and drop the bitrate, and it's fine.

 

Here's what I would do:

Start playback onto your Roku Ultra, then check system performance on the Plex Server. Look specifically at CPU and HDD utilization. It could be that it's transcoding to the Roku, and the transcoding might be too taxing.

For Sale: Meraki Bundle

 

iPhone Xr 128 GB Product Red - HP Spectre x360 13" (i5 - 8 GB RAM - 256 GB SSD) - HP ZBook 15v G5 15" (i7-8850H - 16 GB RAM - 512 GB SSD - NVIDIA Quadro P600)

 

Link to comment
Share on other sites

Link to post
Share on other sites

The testing was done on a windows machine with an i7 4770, 16gb ram, and the file was stored on an SSD. I also noticed the same problem on our older roku 2, but assumed the issue stemmed from being old.

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×