Linking multiple Unraid servers like the Petabyte Project

[wolfslab]

I'm currently planning out a film server for my school and I want to be able to just buy a same spec server and just toss it into the rack and have that just add to the pool so its all the same share like how linus did for Petabyte project. But I am not very familiar with Unraid and there is almost no documentation on this. Does anyone know how this would be setup and have a tutorial on it? We would like this option so when we run out of storage, we can just buy another server and add more storage with little configuration.

 

Specs:

CPU: Ryzen 5 3600

RAM: 16GB DDR4

Motherboard: MSI B450 Pro MAX

HDD: 240gb SSD Cache, 8x 8tb IronWolf NAS drives

HBA: 9207-8i

[Firewrath9]

Don't use consumer gear for servers. You don't get IPMI/KVM, and they're not really designed to be running 24/7.

Also, get ECC as the OS is loaded onto ram, and if theres an error, everything might crash.

[wolfslab]

15 minutes ago, Firewrath9 said:

Don't use consumer gear for servers. You don't get IPMI/KVM, and they're not really designed to be running 24/7.

Also, get ECC as the OS is loaded onto ram, and if theres an error, everything might crash.

Problem is, we don't have much more wiggle room in our budget for enterprise grade hardware. We would like to maybe have a transcode VM running on it as well, so many cores is a must. And there will be an offsite backup as well. And it seems like Ryzen does support ECC, so I can throw in some ECC then.

[Firewrath9]

1 minute ago, wolfslab said:

Problem is, we don't have much more wiggle room in our budget for enterprise grade hardware. We would like to maybe have a transcode VM running on it as well, so many cores is a must. And there will be an offsite backup as well. And it seems like Ryzen does support ECC, so I can throw in some ECC then.

You can find cheap server gear, from 4-5 years ago. Still good stuff, servers are designed to be running for 24/7 at max use for 10 years or more.

A 8c16t (e5-2650 v2) processor is 50$ on ebay, or 100$ for a pair. 16c32t of 3ghz ivy bridge is still quite good.

A dual LGA-2011 board is like 150$, for 2 slots, 16 dimm slots, and a bunch of PCIe and Sata

Heatsinks come included on the mobo, strap on a couple of 60mm fans on them

ECC DDR3 is dirt cheap. 8GB is around 10$, so for the price of 16GB of DDR4 (50$), you can get 40GB of ECC. I'd reccommend getting 8x8, or 4 for each processor for 80$.

 

100+150+80 = 330$ for 16c32t, 3ghz, server features such as IPMI, and ECC, and 64GB of ram.

a 2600+B450+16GB DDR4 is ~120+115+50$.

The ram alone makes it worth it. Also, on a B450 board you won't have 8 sata ports.

 

You can run 6 dual core hyperthreaded VMs with 8GB of ram each, a 2c hypervisor, and leave 2c for unRaid.

Much better than a 2600.

 

Check on reddit.com/r/homelab or reddit.com/r/homelabsales

[wolfslab]

7 minutes ago, Firewrath9 said:

You can find cheap server gear, from 4-5 years ago. Still good stuff, servers are designed to be running for 24/7 at max use for 10 years or more.

A 8c16t (e5-2650 v2) processor is 50$ on ebay, or 100$ for a pair. 16c32t of 3ghz ivy bridge is still quite good.

A dual LGA-2011 board is like 150$, for 2 slots, 16 dimm slots, and a bunch of PCIe and Sata

Heatsinks come included on the mobo, strap on a couple of 60mm fans on them

ECC DDR3 is dirt cheap. 8GB is around 10$, so for the price of 16GB of DDR4 (50$), you can get 40GB of ECC. I'd reccommend getting 8x8, or 4 for each processor for 80$.

 

100+150+80 = 330$ for 16c32t, 3ghz, server features such as IPMI, and ECC, and 64GB of ram.

a 2600+B450+16GB DDR4 is ~120+115+50$.

The ram alone makes it worth it. Also, on a B450 board you won't have 8 sata ports.

 

You can run 6 dual core hyperthreaded VMs with 8GB of ram each, a 2c hypervisor, and leave 2c for unRaid.

Much better than a 2600.

 

Check on reddit.com/r/homelab or reddit.com/r/homelabsales

I'll see if we can buy stuff from eBay. Not sure if they will let us. But I need to know how exactly Unraid would be setup so that I can just buy an identical server and any additional servers I add into the rack, just go into the share and expand it, as the Petabyte project does.

 

Also, I did forget to add the HBA card into the original post. Its added now.

[mariushm]

See Refurbished: Dell PowerEdge R520 8-Bay 3.5" LFF Server - UNIXPlus.com

 

They'll be more "legit" than eBay can give you invoice etc

 

500$  for

 

Dell 2U R520 8-Bay LFF PowerEdge Server
1x R520 Mother Board
- Integrated Dual Intel 1000BASE-T Ports
Intel Xeon E5-2440 6-Core 2.4GHz
8x 3.5” Dell Drive Blanks (base configuration)
32GB DDR3 (base configuration)
1x PERC H710 Raid Card
750 Watt Power Supply

 

make sure to ask if you can insert SATA drives into it, i'm not 100% sure. I see SAS cables on the motherboard.

 

You can also buy HBA cards with external ports and buy cables to connect your server to another container that holds only your drives.

For example either build a server in this, or just connect cables from external controller to this case : https://www.newegg.com/p/N82E16811219038

 

[wolfslab]

Could I use GlusterFS with Unraid servers? Thats what seems to be what allows Linus to just slide in more servers and expand.

[brwainer]

55 minutes ago, wolfslab said:

Could I use GlusterFS with Unraid servers? Thats what seems to be what allows Linus to just slide in more servers and expand.

I am fairly sure that the actual Petabyte Project runs on a more traditional linux distribution like CentOS, on which you can run GlusterFS or Ceph. These are what allow expandability across many servers. But this has a cost in terms of CPU and RAM usage. Running “Hyper-converged” where you have VMs running on the same servers as Ceph or GlusterFS is not recommended without much more RAM than you have (I’ve seen 64 GB as the minimum for testing, and 128GB as the minimum for production use). Also, while you could run VMs on that CPU, I wouldn’t try to run a transcoding VM on the same system as Ceph or GlusterFS because they will compete. I expect you would have to throttle the VM to the point that it isn’t worth using it.

[2FA]

Please oh please do not use UnRAID for anything that isn't personal use. It's a security nightmare since so much stuff is ran as root, it's also not that great for multiple users reading and writing to it at once since you will get bottlenecked by the performance of a single drive.

 

1 hour ago, wolfslab said:

Could I use GlusterFS with Unraid servers? Thats what seems to be what allows Linus to just slide in more servers and expand.

GlusterFS on a regular Linux distro is the optimal solution and you're not paying UnRAID license costs (since cost seems to be an issue).

[wolfslab]

17 minutes ago, 2FA said:

Please oh please do not use UnRAID for anything that isn't personal use. It's a security nightmare since so much stuff is ran as root, it's also not that great for multiple users reading and writing to it at once since you will get bottlenecked by the performance of a single drive.

 

GlusterFS on a regular Linux distro is the optimal solution and you're not paying UnRAID license costs (since cost seems to be an issue).

Is there any good tutorials on how to setup a storage server in linux using GlusterFS? I need to be able to access it from any part of the network from both PC and Mac, as well as remote access. Maybe a Nextcloud plugin? Just something that can also have a google drive like interface to access the files remotely. I currently have a 12tb server running FreeNAS on an old server using Nextclould as the primary way to interface with it. I'm sure thats very insecure, but it works for right now. The specs will be what I'll buy (or enterprise grade hardware if I can get it approved). People that don't know much about computers will be accessing it, so it needs to be simple to connect to remotely.

[2FA]

1 hour ago, wolfslab said:

Is there any good tutorials on how to setup a storage server in linux using GlusterFS? I need to be able to access it from any part of the network from both PC and Mac, as well as remote access. Maybe a Nextcloud plugin? Just something that can also have a google drive like interface to access the files remotely. I currently have a 12tb server running FreeNAS on an old server using Nextclould as the primary way to interface with it. I'm sure thats very insecure, but it works for right now. The specs will be what I'll buy (or enterprise grade hardware if I can get it approved). People that don't know much about computers will be accessing it, so it needs to be simple to connect to remotely.

You would need to use command line for it, so if that's a negative, then it's not for you. There's probably not going to be a tutorial that exactly fits your exact use case but consider reading through some of the official documentation (https://docs.gluster.org/en/latest/) to learn some of the concepts. You could use a software RAID such as mdadm or LVM and use that for the underlying "brick" of the Gluster volume or have Gluster manage all the individual disks with each being a "brick." If you go with the former, a simple Distributed Gluster volume would work for you (where it simply pools the capacity of each server, similar in idea to JBOD), whereas in the latter a Distributed Dispersed volume would work better (kind of similar to RAID striping and each server would be a subvolume of the Gluster volume).

 

All of that is stuff only you will see, you could then install Nextcloud and configure it to use the Gluster volume as the storage directory.

[Nick7]

Save yourself a lot of hassle with what you intend to do.

If I were you, and intend to add lots of drives in future, I'd just get enclosure like: this one or something similar.

[wolfslab]

7 hours ago, Nick7 said:

Save yourself a lot of hassle with what you intend to do.

If I were you, and intend to add lots of drives in future, I'd just get enclosure like: this one or something similar.

Would an adapter like this be compatible with unraid? I'm trying to get a Xeon server, but in the event I can't, I do have to use Ryzen until we can upgrade to Xeon and I cant find any 3rd gen Ryzen boards that have legacy PCI. This card would be to adapt the MFU SAS 9207-8i HBA Card.

https://www.amazon.com/StarTech-com-PCI-Express-Adapter-Card/dp/B0024CV3SA

[mariushm]

You can find 8 port HBA cards on eBay for 30-50$ ... it's really not worth buying such adapters.

PCI limits you to 133 MB/s while a 30-50$ HBA card runs at pci-e x4 , so you get up to 2 GB or 4GB/s to the hard drives connected to it.

 

internal:

Refurbished: LSI SAS 9211-8i 8-port 6Gb/s PCI-E Internal HBA RAID - UNIXPlus.com

Refurbished: LSI 9210-8i 8-port 6Gb/s SAS-2 ZFS JBOD IT-Mode HBA - UNIXPlus.com

 

external

Refurbished: LSI 9207-8e 8-port 6Gb/s SAS+SATA External ZFS JBOD IT-Mo - UNIXPlus.com

 

They're cheaper on eBay

 

 

 

[wolfslab]

1 minute ago, mariushm said:

You can find 8 port HBA cards on eBay for 30-50$ ... it's really not worth buying such adapters.

PCI limits you to 133 MB/s while a 30-50$ HBA card runs at pci-e x4 , so you get up to 2 GB or 4GB/s to the hard drives connected to it.

 

internal:

Refurbished: LSI SAS 9211-8i 8-port 6Gb/s PCI-E Internal HBA RAID - UNIXPlus.com

Refurbished: LSI 9210-8i 8-port 6Gb/s SAS-2 ZFS JBOD IT-Mode HBA - UNIXPlus.com

 

external

Refurbished: LSI 9207-8e 8-port 6Gb/s SAS+SATA External ZFS JBOD IT-Mo - UNIXPlus.com

 

They're cheaper on eBay

 

 

 

Will these work out of the box with Unraid, or do I need to flash it to IT mode?

 

*edit*

Ah, i see that it comes flashed already.

[mariushm]

Just now, wolfslab said:

Will these work out of the box with Unraid, or do I need to flash it to IT mode?

You can select from that combo box to get the version that's flashed in IT mode. The models above are super common, found everywhere, so they should be supported by Unraid just fine.

 

and btw... is your card really classic PCI, or are you referring to a card that's pci-e x4 or pci-e x8 ?

You can install pci-e x4 or x8 card into pci-e x16 slots.

If the Ryzen motherboard only has a pci-e x1 or pci-e x4 and your card is pci-e x8 you also have the option of using a pci-e riser cable to convert the x4 slot into x16 or to a x4 that doesn't have the end wall, allowing you to insert the longer card into the slot.

 

pci-e is designed in such a way that you can insert a card into a smaller slot and have part of the edge connector floating in the air and the card will simply use the number of lanes available in the slot

[wolfslab]

Just now, mariushm said:

You can select from that combo box to get the version that's flashed in IT mode. The models above are super common, found everywhere, so they should be supported by Unraid just fine.

 

and btw... is your card really classic PCI, or are you referring to a card that's pci-e x4 or pci-e x8 ?

You can install pci-e x4 or x8 card into pci-e x16 slots.

If the Ryzen motherboard only has a pci-e x1 or pci-e x4 and your card is pci-e x8 you also have the option of using a pci-e riser cable to convert the x4 slot into x16 or to a x4 that doesn't have the end wall, allowing you to insert the longer card into the slot.

 

pci-e is designed in such a way that you can insert a card into a smaller slot and have part of the edge connector floating in the air and the card will simply use the number of lanes available in the slot

The original HBA card I had selected I thought it was a legacy PCI slot, but was actually a PCIe slot. The one I originally selected is below. As far as I know, this should work out of the box, no flashing, correct?

https://www.amazon.com/MFU-9211-8i-8-Port-Express-Adapter/dp/B07KT4XCM8/ref=sr_1_2_sspa?keywords=9207-8i&qid=1573746192&sr=8-2-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEyWEc2MDJSOE5TUlFXJmVuY3J5cHRlZElkPUEwMTU2MzAwMVE5VjJRSFM0UzZXRSZlbmNyeXB0ZWRBZElkPUEwMzE0MjIzNDlVMVFHTFhLS1BZJndpZGdldE5hbWU9c3BfYXRmJmFjdGlvbj1jbGlja1JlZGlyZWN0JmRvTm90TG9nQ2xpY2s9dHJ1ZQ==

[mariushm]

Yeah that's pci-e x8  ... you can plug it in pci-e x8 or pci-e x16 slot ....

 

You need to use breakout cables with it, which split the SAS connector into 4 x SATA, here's an example: https://www.amazon.com/Cable-Matters-Internal-SFF-8087-Breakout/dp/B012BPLYJC/

It's 8 ports, so you need two cables.

 

It will also work in pci-e x1 or pci-e x4 slots, if you use a riser cable that doesn't have the wall at the end of the connector but you'll be limited to 500 MB/s (for pci-e x1) or 2 GB/s (for pci-e x4) ,

ex of x4 riser which allows longer cards: https://www.amazon.com/ADT-Link-Extension-Extender-Conversion-Vertical/dp/B07THWKR68/

 

or you use one of those pci-e x1 to x16 adapters only you'd be limited to 500 MB/s (because it's pci-e 2.0 card, so that means 500 MB/s for each lane, which means pci-e x1 = 500 MB/s)

https://www.amazon.com/Dr-meter-6-Pack-Powered-Adapter-Extension/dp/B078WLF5F3/