ZFS and GlusterFS on the cheap

[Renegate]

Watching through Linus videos about Petabyte project i thought that the solution of ZFS over GlusterFS is an amazing and really good for scaling idea. I wanted to dive in this project and then i saw this video from Linus expaling how to do it on the cheap 

 Even though it seems tempting and really affordable i want to go with ZFS and GlusterFS for learning purposes. Since it will be hosted in my house i wanted it to be as sillent as possible sto i found a company in England since i live in Europe that makes cases ( www.xcase.co.uk ) so i was thinking of going for a 3U or even 4U chassis so that i could have decent airflow and still be able to have low noise with some Noctua. I already have a 42U cabinet so that's the reason i'm thinking of going for rack servers and it has it's own room so some noise is ok due to the door closing on that room. 

 

My 1rst thoughts was to go with 2 or 3 Dell r710 that are durty cheap now around 250-350 euro but with no disks in them. Also the noise on those was too high even with a closed door and to make matters worse the power consumption was at least 300 Watts each as i saw on a video. Even more troubling was the fact that Linus said that cheap raid controllers don't give proper smart readings on raid configuration and since the video was based on Unraid there was no reference what happens if you choose ZFS with those cards. 

 

On the cpu motherboard side i thought to go with the dual e5-2670 combo as it has the sweat spot for power efficiency and horse power to make it a good addition to almost any purpose that will arise.  

 

I try to dive in the linux industrial uses and i need this to run some labs and get some experience. But since it has this great potential i wouldn't mind using it at home too for some Plex or VM uses. My storage demands in volume size is not much and i'm planning to use new drives on this since it will hold my warm data. Cold storage will propably be a synology so i can tolerate a missconfiguration that breaks the arrays.

 

Any ideas here? My budget is around 2000-3000 for the servers.

 

[unijab]

How many servers do you want and how much storage in each?

[leadeater]

It's GlusterFS btw, not ClusterFS.

 

3 hours ago, Renegate said:

On the cpu motherboard side i thought to go with the dual e5-2670 combo

Way more than what is required for this but if it's cheap enough go for it.

[Renegate]

1 minute ago, leadeater said:

It's GlusterFS btw, not ClusterFS.

Sorry for that! I corrected it.

 

9 minutes ago, unijab said:

How many servers do you want and how much storage in each?

I think 3 is a sweat spot and i believe total of 10TB would be fine to test and be able to have a backup.

[Renegate]

6 minutes ago, leadeater said:

It's GlusterFS btw, not ClusterFS.

 

Way more than what is required for this but if it's cheap enough go for it.

I believe it's really nice at 80 euro per CPU.

[leadeater]

3 hours ago, Renegate said:

My storage demands in volume size is not much and i'm planning to use new drives on this since it will hold my warm data. Cold storage will propably be a synology so i can tolerate a missconfiguration that breaks the arrays.

Very good idea, it's quite annoying having to wait for a backup to complete if you want to make a potentially destructive change if you don't have regular backups setup.

[leadeater]

Btw have a look at other technology like this: Ceph, Lustre etc.

 

OpenStack and Ceph have native connectors so you can host you VMs easily in a Ceph storage cluster. Ceph is more complex than Gluster to setup though but still not that hard.

[Renegate]

4 minutes ago, leadeater said:

Btw have a look at other technology like this: Ceph, Lustre etc.

 

OpenStack and Ceph have native connectors so you can host you VMs easily in a Ceph storage cluster. Ceph is more complex than Gluster to setup though but still not that hard.

From educational approach, taking into consideration maintenance, knowledge gained and time spend, what is the best choise to start with from these technologies? I'm adept at linux but in cloud my skills are almost zero. I got good theoritical cloud background but only that. What would you recommend starting with?

[leadeater]

7 minutes ago, Renegate said:

From educational approach, taking into consideration maintenance, knowledge gained and time spend, what is the best choise to start with from these technologies? I'm adept at linux but in cloud my skills are almost zero. I got good theoritical cloud background but only that. What would you recommend starting with?

Gluster is by far the easiest, btw you don't have to use ZFS with it but it does give you features that aren't in Gluster but are in things like Ceph. I'd stick with Gluster and then try out Ceph using VMs.

[Renegate]

5 minutes ago, leadeater said:

Gluster is by far the easiest, btw you don't have to use ZFS with it but it does give you features that aren't in Gluster but are in things like Ceph. I'd stick with Gluster and then try out Ceph using VMs.

How many nodes does Ceph need for a basic deployment?

 

[leadeater]

4 minutes ago, Renegate said:

How many nodes does Ceph need for a basic deployment?

 

Two, but you can actually force it online with one.

[Renegate]

Does anyone have any experience with the Quanta motherboards? I see i can get a dual 2011 socket with 150 euro along with 160 for 2 e5-2670 seems too good to be true! 

[leadeater]

2 minutes ago, Renegate said:

Does anyone have any experience with the Quanta motherboards? I see i can get a dual 2011 socket with 150 euro along with 160 for 2 e5-2670 seems too good to be true! 

Give me the link so I can buy it, j/k 

[unijab]

I priced up an lga1151 asrock mini-ITX , a Celeron 3.5Gz, 1x 16GB ECC ram, 9207-8i, ...is about 600 USD,

 

A case, psu , and hard drives will certainly add to that though.

[Renegate]

5 minutes ago, unijab said:

I priced up an lga1151 asrock mini-ITX , a Celeron 3.5Gz, 1x 16GB ECC ram, 9207-8i, ...is about 600 USD,

 

A case, psu , and hard drives will certainly add to that though.

If you look on a previous answer you will see i found a 2 Xeon e5-2670 with quanta 2011 motherboard and 16GB ram at 370 euro (414 USD). The sas adapter is what i'm asking what to choose for this setup so that i don't lose smart reading on Gluster or Ceph. Will the LSI 9211-8i be ok with all this?

[leadeater]

5 minutes ago, Renegate said:

If you look on a previous answer you will see i found a 2 Xeon e5-2670 with quanta 2011 motherboard and 16GB ram at 370 euro (414 USD). The sas adapter is what i'm asking what to choose for this setup so that i don't lose smart reading on Gluster or Ceph. Will the LSI 9211-8i be ok with all this?

Yes a LSI 9211 is an excellent choice

[Nikolithebear]

Id recommend a couple of R410s and flash the SAS6i card to IT mode then its basically an HBA.(Maximum of 2tb drives)   Plus the r410 is "fairly" quiet depending on room temp and REALLY cheap with plenty of horse power. I already have 1 of the but Im working on picking up a second one when the time is right. 

Below are some good file server's from Dell for a fairly cheaper then building your own.  

http://www.ebay.co.uk/itm/Dell-PowerEdge-R510-12-Core-2-26GHz-L5640-64GB-12x-Trays-H700-1-YR-WTY-12B-EE-/381931325876?hash=item58ece189b4:g:CJAAAOSwo4pYgisf

 

http://www.ebay.co.uk/sch/i.html?_from=R40&_sacat=0&LH_BIN=1&_sop=2&_nkw=Dell r410&_dcat=11211&Memory%20%28RAM%29%20Capacity=16GB&rt=nc&_trksid=p2045573.m1684

[Renegate]

40 minutes ago, Nikolithebear said:

Id recommend a couple of R410s and flash the SAS6i card to IT mode then its basically an HBA.(Maximum of 2tb drives)   Plus the r410 is "fairly" quiet depending on room temp and REALLY cheap with plenty of horse power. I already have 1 of the but Im working on picking up a second one when the time is right. 

Below are some good file server's from Dell for a fairly cheaper then building your own.  

 

dell r410 are not that much quiet. I got a r210 and at idle the whole house can hear the fans! It's really hard to keep the room cool because we get 40C degrees here in the summer so i need an all season solution. The r510 on the other hand is a bit more quiet but i think there are better solutions with more power efficiency. I don't need it to be 1u or 2u. I would be fine with 3u and 4u chassis!