LTT Storage Rankings

[Eric1024]

Yay, another Linux/ZFS build, and in a Fractal case as well! :)

I wouldn't exactly say Linux Mint is a weird OS choice. From what I've seen and read

it seems to be an excellent distro, well suited for beginners and advanced users as

well if they so choose (I'm running Arch, but that's a different story ;) ).

If I may ask: Why 32 GB of RAM? Are you running deduplication on the ZFS pool? That

uses a crapload of RAM, so if you're not running it yet it might actually be worth

looking into if you have some data which can be deduplicated (I don't, plus I only

have 4 GB of ram, about a third of which is usually used by ZFS).

Anyway, awesome little machine you have there. :)

Thanks for the kind words  :)

 

And yeah mint has been very awesome for me. It's based on Ubuntu so basically everything that works on Ubuntu works on mint (at least in my experience so far) and I like the GUI a LOT better than Ubuntu's GUI.

 

And 32GB of ram for several reasons:

 

One: My workstation originally had 8x8GB dimms in it and I never used more than 20GB or so, so I basically just stole 4 dimms from my workstation.

 

Two: More Cache = more better. I plan to run VMs from my workstation that are hosted from my server, so the extra ram to cache the OS's of those VMs.

 

Three: Breathing room for lots of servers (running modded minecraft servers or bukkit servers for map creation can take a TON of ram... a buddy of mine and I were dicking around in world edit and we had the server chewing down ~ 10GB of ram  :blink: )

 

And also important to note, according to a guide on zfsonlinux.org, to run deduplication effectively, you need about 3.35GB of ram per TB of storage... so with only 4TB or so filled I would need 14GB of ram or so just for dedupe tables, and I want about 8-10GB free for servers, which would only leave about 8GB for the L1 cache, which isn't quite enough IMO, especially when you consider the fact that the more storage I fill up the larger that dedupe table will get, and after I get about 2/3 full the table alone would be 28GB... which leaves a mere 4GB for servers, cache, and the OS.

 

Next time I build a server I think it's going to be a 2P system with probably 4-6x as much memory.

 

 

@Eric1024 nice build, added to the list with 18TB  :) 

 

Thanks :D

[alpenwasser]

Two: More Cache = more better. I plan to run VMs from my workstation that are hosted from my server, so the extra ram to cache the OS's of those VMs.

Three: Breathing room for lots of servers (running modded minecraft servers or bukkit servers for map creation can take a TON of ram... a buddy of mine and I were dicking around in world edit and we had the server chewing down ~ 10GB of ram :blink: )

And also important to note, according to a guide on zfsonlinux.org, to run deduplication effectively, you need about 3.35GB of ram per TB of storage... so with only 4TB or so filled I would need 14GB of ram or so just for dedupe tables, and I want about 8-10GB free for servers, which would only leave about 8GB for the L1 cache, which isn't quite enough IMO, especially when you consider the fact that the more storage I fill up the larger that dedupe table will get, and after I get about 2/3 full the table alone would be 28GB... which leaves a mere 4GB for servers, cache, and the OS.

That makes sense. I didn't recall exactly how much RAM was recommended for dedup, I

just remembered that it was a lot. And for VMs and such lots of RAM certainly isn't a

bad idea.

Next time I build a server I think it's going to be a 2P system with probably 4-6x as much memory.

Now that is a build log I want to see. :D

[Eric1024]

Now that is a build log I want to see. :D

You and me both man... it's in the long term plans, so give it a few years, but in the mean time it's just a daydream... ya'know, 2 broadwell xeon e5's... 8x16GB ddr4 2133... sigh just a dream for now  :wub:

 

But if and when something like that happens, I will definitely make a build log with lots of pictures...

 

Oh and btw, your builds are awesome, I was looking through your logs the other day  :D

[alpenwasser]

You and me both man... it's in the long term plans, so give it a few years, but in the mean time it's just a daydream... ya'know, 2 broadwell xeon e5's... 8x16GB ddr4 2133... sigh just a dream for now  :wub:

Oh dear god! :wub:

I have actually been contemplating making another storage build, something a bit more

beefy than ZEUS. The main restriction with ZEUS was that it had to fit into the sideboard

in our living room, so I couldn't just use an enormous case for it with space for lots

of disk drives. I'm very happy with the build, but I only have ~1.6 TB of free space left

of the 12 which are available for storage.

I wouldn't need to spec it as high as yours because I really only would be using it for

storage (so it would basically just need to be able to run ZFS), so something like an i3

would probably suffice (I don't have high I/O and bandwidth requests).

But yeah, that's also a very long term project, I'm going back to college in fall and

pretty much most of my money has either been spent on ZEUS and HELIOS or is reserved

for my education (not that I'm complaining about that ;)).

Oh and btw, your builds are awesome, I was looking through your logs the other day  :D

Thanks, I appreciate that! :)

There should be an update on HELIOS this weekend, I have finished the primary power

cabling today.

[Eric1024]

Oh dear god! :wub:

I have actually been contemplating making another storage build, something a bit more

beefy than ZEUS. The main restriction with ZEUS was that it had to fit into the sideboard

in our living room, so I couldn't just use an enormous case for it with space for lots

of disk drives. I'm very happy with the build, but I only have ~1.6 TB of free space left

of the 12 which are available for storage.

I wouldn't need to spec it as high as yours because I really only would be using it for

storage (so it would basically just need to be able to run ZFS), so something like an i3

would probably suffice (I don't have high I/O and bandwidth requests).

But yeah, that's also a very long term project, I'm going back to college in fall and

pretty much most of my money has either been spent on ZEUS and HELIOS or is reserved

for my education (not that I'm complaining about that ;)).

Thanks, I appreciate that! :)

There should be an update on HELIOS this weekend, I have finished the primary power

cabling today.

Yeah an i3 would be more than fine, even with compression on (which I would recommend if you're doing any kind of backups). Maxing out gigabit ethernet uses less than 5% of my i7. I would highly recommend that you use an ssd for the boot drive though so that you can partition some of it for the ZIL, because having the ZIL on a mechanical drive can apparently cause serious slow downs, but that's just from what I've read.

 

Looking forward to the HELIOS update :D 

[looney]

o0 can wait for HELIOS update :p @alpenwasser

[8521546]

Why doesnt it reach this list if its a work/gaing pc with 10tb + of storage?

[looney]

8521546, on 08 Jul 2013 - 12:19 AM, said:

Why doesnt it reach this list if its a work/gaing pc with 10tb + of storage?

Good question, there is not really a reason anymore, the old topic was very focused on servers, you also had to have RAID for example, but that criteria is gone now.

Anyway, the criteria are not set in stone, so everyone who thinks we should remove the server criteria, please like 8521546´s reply if you agree with him or reply with a post saying you agree with the criteria change.

but i don´t think there are a lot of 10TB+ game/workstations, seems a bit silly not to get a server at that point

[MG2R]

Why doesnt it reach this list if its a work/gaing pc with 10tb + of storage?

If you read the original post, you (well, I did at least) also get the idea that this thread is meant to give people getting into home servers and redundant storage solutions an idea of what is possible.

[looney]

If you read the original post, you (well, I did at least) also get the idea that this thread is meant to give people getting into home servers and redundant storage solutions an idea of what is possible.

your choice or words is much better then mine, the only reason I removed the redundancy part was because there where some sick servers that dint use raid on all the drives and it would be a shame if they weren't listed.

But either way, if a majority turns out to be against the server criteria I might change it.

[MG2R]

your choice or words is much better then mine, the only reason I removed the redundancy part was because there where some sick servers that dint use raid on all the drives and it would be a shame if they weren't listed.

But either way, if a majority turns out to be against the server criteria I might change it.

I think it would be appropriate adding workstations that have a very specific storage setup. Sometimes, you'll see systems where the storage solution is more advanced/specialised in a workstation than in a home server. If it's just a workstation with 10TB worth of hard drives, however, I think that this isn't the thread to put them in.

 

Of course, the choice is yours.

[looney]

problem there is how to define a "very specific storage setup".

and also, if someone where to post something truly awesome that does not make it on the list ill just put him in your list :P

I think I´ll just leave it as it is, people can post their system anyway, if it turns out that there are some cool work/game-station storage build out there then Ill edit the list, but like I said, I don´t think there will be many, especially if they have to be privately owned.

[LogicalDrm]

Why would anyone have 10TB+ drivespace without raid? If I woul to get so much storage, it would be obvious step. Atm I'm nowhere near that point, getting just over 2TB soon.

[looney]

Why would anyone have 10TB+ drivespace without raid? If I woul to get so much storage, it would be obvious step. Atm I'm nowhere near that point, getting just over 2TB soon.

well, @raphaex system has 20TB worth of JBOD for instance, and I have around 7TB worth of JBOD that I use for iSCSI.

[LogicalDrm]

@looney, I assume those aren't same as raid? I'll google them when I get out of work.

[looney]

@LoGiCalDrm

JBOD? Just (a) Bunch Of Drives, so no RAID

 

iSCSI? i'ts a protocol that allows you to connect a drive over the network, I share the drive as a iSCSI target on my server, and then I connect it to my laptop for instance, from a OS perspective it will act just like a normal HDD, even though its actually in my server.

[D.Dominic]

How do you manage to fill up more than 5TB of data?!?!?!?!

[looney]

How do you manage to fill up more than 5TB of data?!?!?!?!

well, if you're only concerned about "filling the drive up" you can use true crypt to make a 5TB encrypted file and your done :p

[D.Dominic]

well, if you're only concerned about "filling the drive up" you can use true crypt to make a 5TB encrypted file and your done :P

Do you personally need that much storage? Just really curious here XD

[Eric1024]

Do you personally need that much storage? Just really curious here XD

Oh it's easy to fill up that much space. Between backing up a few computers and storing a few hundred movie and tv show rips I'm already at 4TB. With a much larger library, you can hit 20TB easily. 

[looney]

Do you personally need that much storage? Just really curious here XD

Not really probably, especially with fiber, but I just really like storage :P

 

I started with 1 2TB NAS 3 years ago, I upgraded that one to a 4TB NAS in less then a week.

Then I get a second 4TB NAS, after thatI switched to a 14TB server (1,5 years ago).

And now i'm on my second server (CPU, RAM and HDD's are mostly from the old server)

 

And when this one is full I'll use a SAS expander to add another case to my server.

[alpenwasser]

How do you manage to fill up more than 5TB of data?!?!?!?!

Easy. As soon as you start storing substantial amount of High-Def content you will

soak up the TB's very quickly.

Even more so if you make videos yourself, in which case you will have lots of rather

badly compressed video content on your drives. One .mov file from my DSLR is ~ 360 MB

per minute of video. Of course I could delete those files after finishing and compressing

the final product, but since compression in video is lossy (usually) I would loose

quite a bit of information from the original. If I ever wanted to make a higher quality

version of the video, or simply use the clip for another project, that could quite

noticeably impact video quality.

EDIT:

Then there's also large picture libraries. I have written a few simple web crawling

scripts which scour certain websites for pictures and download them in the highest

possible version available automatically. On some sites, this means .TIFF files of

maybe several GB in size. Since the process is automated I can (and have) easily

download(ed) several thousand pictures with very little effort. A good example of

such pictures are astronomy sites, such as Hubble and the European Southern Observatory.

[D.Dominic]

Easy. As soon as you start storing substantial amount of High-Def content you will

soak up the TB's very quickly.

Even more so if you make videos yourself, in which case you will have lots of rather

badly compressed video content on your drives. One .mov file from my DSLR is ~ 360 MB

per minute of video. Of course I could delete those files after finishing and compressing

the final product, but since compression in video is lossy (usually) I would loose

quite a bit of information from the original. If I ever wanted to make a higher quality

version of the video, or simply use the clip for another project, that could quite

noticeably impact video quality.

EDIT:

Then there's also large picture libraries. I have written a few simple web crawling

scripts which scour certain websites for pictures and download them in the highest

possible version available automatically. On some sites, this means .TIFF files of

maybe several GB in size. Since the process is automated I can (and have) easily

download(ed) several thousand pictures with very little effort. A good example of

such pictures are astronomy sites, such as Hubble and the European Southern Observatory.

Wow never knew how someone could use that much data, thanks for the info!

[alpenwasser]

Wow never knew how someone could use that much data, thanks for the info!

It helps if you have a tendency to not want to delete stuff. It's a bit like not

wanting to throw away old things. ;)

[Eric1024]

It helps if you have a tendency to not want to delete stuff. It's a bit like not

wanting to throw away old things. ;)

Digital hoarding ftw :D