ZFS vs RAID+ZFS

[tekgeek1205]

So ive got 16x  2tb sas2 drives and a couple options in how I can arrange the array. In the past ive been using btrfs on a raid 6 array across 10 disks. I ran out of room and have upgraded....Everything! I've backed everything up and im ready to start clean. I've just got to decide on the layout. Also want to switch to ZFS because that's what LXD requires to use all its features. Here are the options I'm considering:

 

RAID Card LSI 9261-8i +1GB Catch +BBU (there is no IT firmware from my understanding)  

- a single raid 6 across all 16 disks and a span/simple ZFS volume ( or is raid 0 the only option requiring another 16 disks to expand, yikes)

- 2x raid 5 across 8 disks each and raid 0 ZFS (8 disks at a time is much cheaper)  

 

OR ditch the LSI 9261-8i for the LSI 9207-4i4e I use for my LTO5 library and 

-let zfs do its thing

 

 

System Specs and Workload

 

 

System Specs

Chassis : Supermicro SC836TQ-R800B

Motherboard : Supermicro X8DAH+-F

2x W5580 3.2 base, 3.46 max turbo   (plan to upgrade to x5680s)

12GB DDR3 1333 ECC Registered (looking to put 8GB modules in all 6 memory channels)

LSI 9261-8i Raid FW

LSI 9207-4i4e IT FW

2x 120GB SATA SSD for boot and VM's/Containers

16x Seagate ST320004CLAR2000 (7.2k nearline sas2)

Dual 10gbe fiber (one to a Dell PowerConnect 6224 and the 2nd to my desktop, because.....why not)

Quad Gbe (so router vm has dedicated NIC's)

3x 16 port sas2 expanders ( only using one, currently )
 

I use this machine to for a couple different servers:

 

Ubuntu 18.04 Server -this is the base OS running

-KVM

-LXD

-OpenvSwitch

-TFTP Server

-NFS & SAMBA nas

 

 

LXD containers in Ubunu 18.04 Servers

-Jackett

-Sonarr

-Radarr

-qBittorrent

-Handbrake 

 

KVM Virtual Machines

Plex - Ubuntu 18.04 Server

  -6 users

  -2x dual tv tuner cards (4 channels at a time) for DVR and Library Recording @1080p

            (Haven't yet attempted PCIe pass-throughhowth LXD)

pfSense - FreeBSD Based Appliance install 

  -DHCP

  -DNS

  -VPN

  -Squid

  -NTP

Zabbix - Ubuntu 18.04 Server

  -This months monitoring server, IPMI, etc

  - Im still looking for central monitoring sorftware that i like

 

I also spin up new KVM's to try things, or test something before applying it to my active servers.

 

Ive probably forgotten some

 

 

[Electronics Wizardy]

Id just use zfs for raidz. its allows for data  protection with checksumming and its nice to manage all storage in one spot.

 

 

[Hugs12343]

21 minutes ago, Electronics Wizardy said:

Id just use zfs for raidz. its allows for data  protection with checksumming and its nice to manage all storage in one spot.

 

 

Yeah use zfs.

[Salv8 (sam)]

zfs is probably your best option, i think you can just add drives an expand your storage but don't quote me on that as i am unsure if zfs can do that...

[Electronics Wizardy]

1 minute ago, Salv8 (sam) said:

zfs is probably your best option, i think you can just add drives an expand your storage but don't quote me on that as i am unsure if zfs can do that...

You can add drives to a zpool easily, but its striped so if any drive fails you lose data, and you can't remove a drive later on.

 

You can expand a vdev by making all the the drives inside the vdev bigger one at a time aswell.

[Salv8 (sam)]

4 minutes ago, Electronics Wizardy said:

You can add drives to a zpool easily, but its striped so if any drive fails you lose data, and you can't remove a drive later on.

 

You can expand a vdev by making all the the drives inside the vdev bigger one at a time aswell.

is it possable to have 2 zfs's in one system then RAID 1 then for protection of drive failures?

[leadeater]

Don't combine hardware RAID with ZFS, you'll actually lose most of the ZFS data integrity features/effectiveness if you do that. ZFS assumes control over the data is absolute and what it has done is the truth which is not the case if that data is then sitting on hardware RAID.

 

Data is still perfectly safe on hardware RAID and that can do patrol reads to fix data integrity issues but ZFS assumes that is not possible at the file level, it won't corrupt ZFS in any way but only if you have the data spread across more than 1 RAID array as ZFS relies on multiple copies for the integrity it does (or parity chunks).

 

Short version is hardware RAID lies to ZFS and ZFS doesn't like lies.

 

If you have to use hardware RAID in non IT mode create a RAID 0 array for each disk then treat as if they were on an HBA and not a RAID card.

[tekgeek1205]

from what i understand about ZFS is you want any level of falt talerance you need to use sets of raidX vdevs spanned. Kinda like raid 10, 50, and 60, but it 01, 51, 61 effectively. Because you are striping lets say 2 zraid5 arrays to make zraid51. But why use zfs for the first raid layer when i can use hardware raid. Ive replaces a failed drive in with my cards biso. it took all of 10 min +the LIVE rebuild time. 

 

Edit: 

i had not read leadeater's comment befor wrighting this.

[tekgeek1205]

So with a 16 disk array that one want to be able to expand easily, how should i slice things up for ZFS. Lets assume im using HBA's in IT mode. Also, I dont want to be forced to expand the array with another 16 Disk chunk, I can do 8 disk  chunks.

 

So zraid5 with 8 disks - is there an optimal disk count or multiplier?

 

then span or strip them? 

 

am I going down the right path or am i lost?

 

Does ZFS allow for a hot spare?

 

[Electronics Wizardy]

43 minutes ago, Salv8 (sam) said:

is it possable to have 2 zfs's in one system then RAID 1 then for protection of drive failures?

Do you mean a mirrored vdev, cause thats common. A vdev can be a single drive, mirrored, or a raidz(1,2,3). A zpool is filled with vdevs.

[tekgeek1205]

Then I create my volumes in the pool (what i was referring to as the stripe/span set) of raidz1 (what i was calling raid5)?

 

If thats right, I think I understand now.

 

Im going to make 2x raidz1 vdevs and put them in a pool. I then use that pool like i would a LVM and make volumes/partitions. So what the term for the final device i mount, the logical volume. 

 

Edit:

this allows me to expand in 8 disk chunks?

[jde3]

Let ZFS manage your drives directly. This is what it's good at and how it expects to run.

 

You kind of have a nice rig going there but it's just a shame your using Linux when FreeBSD is staring you in the face with it's superior containers and native ZFS and PF support.

 

It's nice to see Ubuntu decided to put on it's big boy pants and use a grown up file system though. Maybe there is hope for Linux yet..

[tekgeek1205]

 

Im i doing somthing wrong? Its sooooo slow. Like single spindle speed. Here's the topology:

 

Quote

:~$ zpool status
  pool: spinningrust
 state: ONLINE
  scan: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        spinningrust  ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            sdd       ONLINE       0     0     0
            sde       ONLINE       0     0     0
            sdf       ONLINE       0     0     0
            sdg       ONLINE       0     0     0
          raidz1-1    ONLINE       0     0     0
            sdh       ONLINE       0     0     0
            sdi       ONLINE       0     0     0
            sdj       ONLINE       0     0     0
            sdk       ONLINE       0     0     0

 

Benchmarking using: sudo time sh -c "dd if=/dev/zero of=/home/ryan/test.tmp bs=8M count=1250 && sync"  and sudo time sh -c "dd if=/dev/zero of=/home/ryan/test.tmp bs=8M count=1250 oflag=dsync"

 

Quote

	2x 4 Disk Raidz1	2 Disk SSD Raid0 LSI 9261-8i	7 Disk HDD Raid6 LSI 9261-8i
	/spinningrust/test/test.tmp	/home/ryan/test.tmp	/storage/downloads/test.tmp
Sync'ed
4k / 6,500,000 / 27GB 25GiB	292	513	555
2k / 13,000,000 / 27GB 25GiB	216	270	261
1k / 26,000,000 / 27GB 25GiB	119	164	174
1M / 10000 / 10GB 9.8GIB	35	394	299
2M / 5000 / 10GB 9.8GiB
Dsync'ed	51.3	439	394
Sync'ed	281	585	624
4M / 2500 / 10GB 9.8GiB
Dsync'ed	75.3	442	462
Sync'ed	198	569	631
8M / 1500 / 10GB 9.8GiB
Dsync'ed
Sync'ed	148	551	625

 

[jde3]

Raidz1 is single spindle read/write, you have them in a stripe so you should be 2x. If you do this in a mirror config you'll get n number of mirrors on read.

 

Not sure sync works properly on ZFS or not (it will re-order all the writes and flush them to disk in a logical manner by default) Also what options did you use creating the pool?

 

You can do this.. but you really don't want to if you can help it. If you loose a disk your resilver is going to be painful and scary. If your looking for performance mirrors are the way to go.

[tekgeek1205]

yeah im reading.....and reading......and reading.....google will get me there soon enough. im going throught all the options with notepad open. im going to try a couple of of different topologies once ive gone through the options. 

 

what should i use to benchmark. DD seams cumbersum and the the only other thing i could find with google and zfs benchmarking was bonnie++ and i find the output hard to read

[jde3]

Your benchmark is ok i just don't know if sync is doing what we think it is. (it may be, I just don't know you can adjust the dataset property sync to always to force sync)

zfs create -o sync=always -o compression=lz4 tank/benchmark

 

ZFS with a decent amount of ram isn't a slow FS but it is doing a lot of work behind the scenes. LZ4 is actually a performance boost on ZFS now with compressed ARC.

[jde3]

Basically you can try to get the most out of the storage and use raidz or you can get iops and use mirrors and lose efficiency but you can't do both. Small groups of mirrors should fly on random reads. your write speed is going to depend on how many vdev groups you are comfortable with.(the more the scarier)

 

Also when you create the pool you may need ashift=12 for your devices. (if they emulate 512b block size) it's usually safe to do, FreeBSD defaults to it now.

[tekgeek1205]

i was using the dsync option to get around the system catches. In doing so i went arround all the ZFS goodies. My next attempt was going to use ashift and LZ4. I'll probably just start with the option set you posted. It was about what i was going to do anyway. I guess i wright a script for dd so i can walk away.

[tekgeek1205]

So i was getting really bad performance. I kept trying different things until i decided to install as the root file system. I followed this guide to the letter. and here and my results:

 

	4x 2 drive mirror ZFS fresh Ubuntu 18.04 Server
	rpool/home/ryan   ashift12
Sync'ed	16 threads
8M / 1500 / 10GB 9.8GiB
	/dev/shm only gets 2300
4M / 2500 / 10GB 9.8GiB	2880
2M / 5000 / 10GB 9.8GiB	2880
1M / 10000 / 10GB 9.8GIB	2880
8K / 6,000,000 /	2320
4k / 6,500,000 / 27GB 25GiB	1760
2k / 13,000,000 / 27GB 25GiB	1280
	CPU @ 100% for everything below 1k
1k / 26,000,000 / 27GB 25GiB	800
512b / 30,000,000 / 15GB 14GiB	400
256b / 6,000,000 / 1.5GB 1.4GiB	208
128b / 6,000,000 / 768MB 732MiB	106

[jde3]

Yep.. mirrors are the way to go. Might not want to use /dev/zero in your test because that will be all compressed to nothing when testing writes. urandom on the other hand would be incompressible so.. neither is fair. To test a read you can use the same method, just send a file to /dev/null... there has to be something better though..

 

That looks like a good guide. It's nice they have the autosnapshots in there.

 

One thing I'd add is to set vm.swappiness to like 5 or 10. in your /etc/sysctl.conf (or similar on ubuntu)

 

Why? When Linux gets under ram pressure it seems to voluntarily move objects to swap instead of removing objects from ZFS's ARC. ZFS uses Solaris's memory system from the SPL. With FreeBSD they are much closer to Solaris so their porting layer is thinner. (Solaris is a descendant of ancient bsd.)

 

Other than that all I can say if if your using ZFS on root on Linux to keep a backup kernel that can boot ZFS installed. For me I usually just make one separate and stick it in my efi or boot dir that way it's there in case some auto-update decides not to include the module some day. ZFS rescue CD's exist also. I believe Funtoo/Gentoo has one. (if your version of ZoL gets too low on the rescue cd though it will import read only) Ubuntu had up until recently a very old version of ZoL, that may hold back your kernel updates if they do that in the future.

 

Like I said originally FreeBSD is just better at this, you don't gain much from using Linux.. but if your life depends on Docker (outside of a VM that is) it's what you'd have to use.

[tekgeek1205]

So whats the learning curve for FreeBSD from linux. Im just now getting a good hand on Ubuntu. Do you think its worth the transition. Ubuntu is the only thing I've used other than Windows, and ESXi. My plex server went from an Windows 10 box with a raid card to a PowerEdge 2950 with ESXi, then to My current setup with Ubuntu. I chose to leave ESXi because of the vCPU limitation with the free license. I put everything in there own VM's at first, then about a month ago I put all I could into LXD containers. 

 

I was useing Kimchi to manage the KVM's and LXD-WEBUI for the containers.  

[jde3]

12 hours ago, tekgeek1205 said:

So whats the learning curve for FreeBSD from linux. Im just now getting a good hand on Ubuntu. Do you think its worth the transition. Ubuntu is the only thing I've used other than Windows, and ESXi. My plex server went from an Windows 10 box with a raid card to a PowerEdge 2950 with ESXi, then to My current setup with Ubuntu. I chose to leave ESXi because of the vCPU limitation with the free license. I put everything in there own VM's at first, then about a month ago I put all I could into LXD containers. 

 

I was useing Kimchi to manage the KVM's and LXD-WEBUI for the containers.  

FreeBSD is a better server but Linux is still a better desktop, so it depends what you want to use them for.. Learn the both?

 

I tend to find FreeBSD a lot easier to manage and to customize. Linux tends to have things that are needlessly complicated. An example is in FreeBSD ifconfig controls your wifi. And you can set it right from /etc/rc.conf. no network manager, iwconfig.. or 3 other programs to do it like in Linux. - You have a program for managing network interfaces and it manages them.. why is this so complicated on Linux? Who knows.. maybe because 100 million different people are trying different things.

 

FreeBSD is well documented and it doesn't have wild divergence with distros. Where as Linux is a kernel and a userland from different groups combined together by distros and 3rd parties, FreeBSD all comes from the same people. It's easy to manage, and yes there are some differences but most of them seem to be positive ones when you use it for a while. FreeBSD is much more careful about making changes than Linux is, so it may seem to lag behind in some areas.. but sometimes when it does it can see the mistakes other OS's make and do the right thing instead..

 

(A userland is the main C Library and collection of core OS tools and programs. your /bin/sh, ls, cp, date, make etc.. The switches from these programs sometimes differ from the GNU ones in Linux. The GNU had to be different, BSD are more traditional Unix'ey. On BSD you'll see gmake, gdate etc if you need specific GNU versions, the shell is also different but you can install bash if you like it.)

 

Pros: It is fast, it's got native ZFS, Jails built in, dtrace, PF firewall and an extremely fast network stack. It also has ports that is similar to Gentoo's Portage (Portage was inspired from Ports, Daniel Robbins was a FreeBSD fan before he made Gentoo)

Cons: It's not Linux so it receives less attention, drivers for bad hardware may be spotty but quality hardware runs well and there are not many closed source binary blobs to run (oh darn). So if your on a Laptop you might be better off with MacOS. (btw MacOS uses the same userland from FreeBSD, so if you know one you know the other.) I think FreeBSD is growing in popularity however. I've used Unix for 20 years and never seen it more popular than it is today. Apple, Sony, Nintendo, Whatapp and Netflix are all based on it. VPS's like Ec2, Azure and Digital Ocean offer images for it. It is still smaller than Linux though.

 

Your on the right track with containers, they are more efficient than VM's. FreeBSD has the oldest container system Jails that has seen a lot of life recently with the container revolution. It's a more solid solution and a safer one as it was made from a security standpoint and not a virtualisation one like docker.. but it works a little different too.  Mostly Jails are managed from the command line (via rc.conf, iocage or easy jail) There are other things in the works however. https://clonos.tekroutine.com/ this is a jail manager (based on cbsd) I've used it and I have to say it's not quite there yet.. but it shows the potential for something like this to appear.

 

If your looking for something more desktop like, GhostBSD is FreeBSD with a few tweaks and default packages installed for desktop use. You'll find most of the same open source programs your use to on Linux. Notable missing things would be steam and other closed source stuff.

[jde3]

Oh yeah, if you like Plex also.. Check out Emby. I've been quite happy with Emby for a media server.

[tekgeek1205]

Ive currently only gotten the base system and NFS/SAMBA running to access my media, once i get my core services backup. I'll create a FreeBSD VM and start to play around. When I decided to leave ESXi I went to the only other thing I had used other than windows, ubuntu.

 

I like the idea of a more controlled ecosystem than Ubuntu. Every time I turn around Ubuntu has changed something. For example, 18.04 dropped the old network config for Netplan. It works great, you must "generate" or compile your config before applying so you know if there is syntax prob before you apply. I guess they were going for a more idiot proof network config, but you have to learn a new markup language, its not compatible with OpenvSwitch, all you old custom scripts that modify the network are now broken, bla, bla ,bla. With the constant changes all the guides are hit and miss.They break things every LTS update, to the point that you have you stay on the old LTS for a year or 2 before things get fixed or attempted to revert there changes ( i had to ditch Netplan AFTER I took the time to learn the markup).

 

Too many people developing in parallel without proper communication, and the abandoned projects.......soooooo many abandoned projects. If i find there is something i need in Linux i can always run it in a VM like i current run PFsense. It would also be nice to have my NAS and Router on the same platform. 

 

 

 

What do you think about Intel optane. They have a 16GB M.2 for 41USD. Looks perfect for a slog device. I've got 2 cheap ssd's I need to liberate from my raid card for L2ARC, after i check their IOPs. They are in a case under a monitor, 3 switches, a large lamp,and a tape library, or I would have done it already.

 

 When I move in 2 weeks it gets to be in a full sized rack with slide out rails. I have also set it up so the entire houses return air for the HVAC is forced from the bottom of the rack to the top before being cooled/heated and recirculated. I CAN'T WAIT!

 

 

 

I'll give Emby a try, i've seen reference to it in the past. Does It have dvr functions. My Plex setup uses hauppauge tuner cards to capture from an antenna I have.

 

PS

Cable free since 2004

[jde3]

4 hours ago, tekgeek1205 said:

It works great, you must "generate" or compile your config before applying so you know if there is syntax prob before you apply.

Oh.. Great.. just like Sendmail from 1980. I seen that they had ip (and that is a stupid name for configuring network interfaces because not all of them do IP) This new thing is news to me.

 

This is my static network config in FreeBSD.

ifconfig_em1="inet 192.168.8.32 netmask 255.255.255.0"

 

One line in rc.conf.. soooo hard and confusing.

 

4 hours ago, tekgeek1205 said:

What do you think about Intel optane.

I don't exactly see the point for servers. You have fast cache, ram, slow cache SDD's and physical, spinning disks. IDK how Optaine fits in there or if we need another layer.

 

I haven't used Live TV on Emby but people say it's fantastic and I wanted to add a device to do it.. (homestar something..)