Jump to content

The number of hard drives supported in a server?

Go to solution Solved by Electronics Wizardy,
30 minutes ago, ZFSinmylungs said:

I see. Like I said - physical space ain't a too-big concern to me - I can be a DIY expert.

HBA-s.. how they work.. so HDD > HBA > motherboard, I would assume.. Any good video/read about HBA-s? (I apologize for being lazy)

128-256 drives? That sounds amazing! Sooo no storage-space(TB/PB) limit? I guess the power consumption per drive would also need to be taken into consideration.

You need sas expanders to get that many drives per sas controller. They often are built into the backplans on sas jbods or drive bacplanes.

 

There is no limit of TB that would limit you here. You can use the biggest drives you can buy today without a issue.

 

How many TB do you need in the server total?

Okay, so.

I am new here, hello.

I got a question, after watching a a few of server-related videos from LTT (for the 2nd time) (I'm not new, I'm basically a LTT consoomer) and I wondered..

Gee, Linus, now that you told me what a HBA is (a card to plug all the hard drives into I think), and how it differs from a RAID controller card, now I wonder ... is there a limit? like.. I do not see like a "max hard drives" in server listings other than well... the physical space for them (also called bays) .. is there any other limit? Like imagine daizy-chaining or probably I'd make a DIY case for them I suppose...

I'd have software raid via OpenZFS (better than Oracle's ZFS) thus - not needing a RAID controller card.. basically RAID 10 or.. just mirrors in a zpool.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

I see. Like I said - physical space ain't a too-big concern to me - I can be a DIY expert.

HBA-s.. how they work.. so HDD > HBA > motherboard, I would assume.. Any good video/read about HBA-s? (I apologize for being lazy)

128-256 drives? That sounds amazing! Sooo no storage-space(TB/PB) limit? I guess the power consumption per drive would also need to be taken into consideration.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

30 minutes ago, ZFSinmylungs said:

I see. Like I said - physical space ain't a too-big concern to me - I can be a DIY expert.

HBA-s.. how they work.. so HDD > HBA > motherboard, I would assume.. Any good video/read about HBA-s? (I apologize for being lazy)

128-256 drives? That sounds amazing! Sooo no storage-space(TB/PB) limit? I guess the power consumption per drive would also need to be taken into consideration.

You need sas expanders to get that many drives per sas controller. They often are built into the backplans on sas jbods or drive bacplanes.

 

There is no limit of TB that would limit you here. You can use the biggest drives you can buy today without a issue.

 

How many TB do you need in the server total?

Link to comment
Share on other sites

Link to post
Share on other sites

Yeah.. I've heard the words "drive back-plane" before :) makes sense.
That's mofo awesome! :D Thank you, I feel inspired.
Well.. the beginning would be 32TB/2 soo 16TB usable. I heard other non-mirror RAID solutions aren't that reliable and are VERY slow when it comes to recovering.. why I will go with mirroring, with only 50% usability. This might be in 1 year or 5 years, but definitely something I want.
Thanks for your help, mister/misses.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

36 minutes ago, ZFSinmylungs said:

I see. Like I said - physical space ain't a too-big concern to me - I can be a DIY expert.

HBA-s.. how they work.. so HDD > HBA > motherboard, I would assume.. Any good video/read about HBA-s? (I apologize for being lazy)

128-256 drives? That sounds amazing! Sooo no storage-space(TB/PB) limit? I guess the power consumption per drive would also need to be taken into consideration.

It depends what you mean by limit really.

 

Yes, power consumption can be a large factor (in the sense that IronWolf drives require 1.8 amps on the 12v rail...so you need to either start up drives separately...slower boots, or have a lot beefier power supply/supply the power in a different way).  The idle on IronWolf's is at about 5 watts as well for 12tb drives, so after it has started it doesn't draw too much power (like 0.12 kWh a day per drive).

 

A big thing can be the vibration and heat though if you have a lot of drives.  I've seen people using consumer drives in large array NAS setups, and I've seen them fail.  You would need enough spacing between them so you can have airflow.

 

If you are talking about theory though, there is a space limit.  If you wanted to have it just appear as one volume (instead of hundreds of drives), the limit is 8 PB...if you are using an older edition of Windows 10 (pre 1709), the limit was 256 TB / volume.  If you ignore linking volumes to a directory, then max volumes would realistically be 26...(208PB of storage).  With that said, you would be breaking the bank by doing that at over 4160 drives (assuming 50TB drives) 20.8 kWh.

 

A note though, to achieve 8PB of store a cluster size of 2MB needs to be chosen...while not realistic, in theory you could fill up the space using 4.3 billion files...like 10,000 text files (of 2kB each) one would expect would consume about 20 MB, but in reality in a cluster size of 2MB would consume 20 GB.  (I could be wrong, someone correct me if I'm wrong...since cluster size I believe defines the smallest file size possible).  It is why choosing the correct cluster size can be important...if you intend to store a lot of little files smaller cluster sizes is more space efficient, but it limits how large the drive could be.  The default actually is 16TB (4KB cluster), so 10k files would be 40MB.

 

Real world example, my programming/cache folder totals ~1 million files (rounding for simpler math), avg file size of 20KiB [20GiB].  If I created a volume that could hold 128TB, those files would consume at least 32 GB.  A volume that could be increased to 8PB, it would be 2TB, 1PB it would be 256GB.

 

So yes there is a limit for the upper bounds.  I am just saying all of this, because there will be someone out there that would create a raid 0 type of array thinking they will just add more space when they need to (and over-provision by selecting the largest cluster size that will allow them to expand the most) without realizing you are sacrificing things by doing so.

3735928559 - Beware of the dead beef

Link to comment
Share on other sites

Link to post
Share on other sites

Hahahaha I do not use Windows, I use GNU+Linux on ZFS.
On ZFS: Maximum 16 Exabyte file size; Maximum 256 Quadrillion Zettabytes storage.
I think you'd have a tricky time counting all those zeros :)

damn.. that is quite a lot of wattage per drive o.o
Are you familiar with the differences between manufacturers of hard drives? 0.12kWh can be a lot when they add up.. but again.. ain't no one got money for petabytes of storage lmao especially if one doesn't live in the USA or smtn.

Thanks for the time and effort for your answer lol you sure are informed to the very details.

P.S. I kinda got insulted by the mentioning of the word "Windows" lol sorry for that :P anyways - gave some interesting info, you might find it impressive and investigate or something.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

58 minutes ago, ZFSinmylungs said:

P.S. I kinda got insulted by the mentioning of the word "Windows" lol sorry for that :P anyways - gave some interesting info, you might find it impressive and investigate or something.

lol, I know more about Windows Server since that is what I primarily worked with.

 

1 hour ago, ZFSinmylungs said:

Are you familiar with the differences between manufacturers of hard drives? 0.12kWh can be a lot when they add up.. but again.. ain't no one got money for petabytes of storage lmao especially if one doesn't live in the USA or smtn.

haha, yea I know the feeling.  It's actually why my NAS doesn't run 24/7 (aside from the noise), 10 drives 24/7 can add up in electricity bills.  To be clear though, that's 0.12kWh/day.  So where I live, $0.141/kWh, it's about 1.7 cents a day per drive ($6.17/year/drive)...although that's for a 5W idle (Actually I use IronWolf's 8TB so it's $9/year/drive).

 

I'm not directly familiar with the others, as I haven't had a need to look at the numbers.  Here are the specs for WD though, along with Seagate

https://www.seagate.com/www-content/datasheets/pdfs/ironwolf-12tbDS1904-9-1707US-en_US.pdf

https://products.wdc.com/library/SpecSheet/ENG/2879-800002.pdf

 

I don't know how to feel about WD's Reds anymore though, they do use less power though than Seagate.  They have gone to listing drives as 5400RPM class (instead of given true RPM...as some run at 7200RPM)...also some versions don't play nicely with ZFS (I think the 8TB+ drives still use CMR though).  Actually I think it's their Red Plus drives are CMR.

3735928559 - Beware of the dead beef

Link to comment
Share on other sites

Link to post
Share on other sites

@wanderingfool2
"lol, I know more about Windows Server since that is what I primarily worked with."
hahah no problem :) I don't blame you, you know your stuff, I know mine, considering that.. I think we're communicating quite well :'D

"haha, yea I know the feeling. It's actually why my NAS doesn't run 24/7 (aside from the noise), 10 drives 24/7 can add up in electricity bills."
Hmm yeah.. unless I will run some kind-of service(s) for other people via the biggest WAN, I'd have it shut down.. or at least the disks.. just have a "DISKS MAIN POWER SWITCH" like some evil scientists XD

"although that's for a 5W idle (Actually I use IronWolf's 8TB so it's $9/year/drive). "
Hmm does NOT sound expensive. muuuuultiple that by a factor of at least 30 - that's already a lot.
Unless you have your own source of (hopefully renewable) power.

"also some versions don't play nicely with ZFS (I think the 8TB+ drives still use CMR though). "
Whoah, whoah, whoah, really? So I shall avoid CMR?

"Actually I think it's their Red Plus drives are CMR. "
Thanks.

By the way, what does the "NAS" stand for in hard drive's specifications? Like I know I heard some are made to have like redundancy or error-checking or what-ever integrated, but as I would use ZFS and stuff - I don't want that as I heard it's either unnecessary or it causes problems?
Or are the NAS ones for server-use like tons of them side-by-side because they are less sensitive to mechanical disturbances aka vibrations?

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, ZFSinmylungs said:

Yeah.. I've heard the words "drive back-plane" before 🙂 makes sense.
That's mofo awesome! 😄 Thank you, I feel inspired.
Well.. the beginning would be 32TB/2 soo 16TB usable. I heard other non-mirror RAID solutions aren't that reliable and are VERY slow when it comes to recovering.. why I will go with mirroring, with only 50% usability. This might be in 1 year or 5 years, but definitely something I want.
Thanks for your help, mister/misses.

Parity raid really isn't that bad, and id probably do that here instead of mirrors. They normally rebuild at the max speed of the drive(or the same speed as a mirror here).

 

Also reliability should be about the same, or better with multiple parity.

 

ALso keep backups

15 minutes ago, ZFSinmylungs said:

By the way, what does the "NAS" stand for in hard drive's specifications? Like I know I heard some are made to have like redundancy or error-checking or what-ever integrated, but as I would use ZFS and stuff - I don't want that as I heard it's either unnecessary or it causes problems?

MOstly marketing, but tler is the big feature, so they time out quicker in a error so raid cards are happy, won't matter with zfs.

 

 

 

Also should handle vibrations from other disks better 

Link to comment
Share on other sites

Link to post
Share on other sites

3 minutes ago, ZFSinmylungs said:

Whoah, whoah, whoah, really? So I shall avoid CMR?

CMR should in theory be better.  WD on some of their older drives used SMR, which that class of drives didn't play well with some ZFS systems.  It was a bit of a controversy at the time.  Here is a video regarding it

https://www.youtube.com/watch?v=8hdJTwaTl8I

 

5 minutes ago, ZFSinmylungs said:

By the way, what does the "NAS" stand for in hard drive's specifications? Like I know I heard some are made to have like redundancy or error-checking or what-ever integrated, but as I would use ZFS and stuff - I don't want that as I heard it's either unnecessary or it causes problems?
Or are the NAS ones for server-use like tons of them side-by-side because they are less sensitive to mechanical disturbances aka vibrations

It's a bit of marketing, but it's also a way to class drives.  Some might have better vibration tolerances, and are more designed to work 24/7.

 

A consumer drive, after all, likely gets parked a lot more and has a quicker spin up (or at least I've noticed that for Seagate drives).  With that said, the lines between certain types of drives seem to be blurred.

3735928559 - Beware of the dead beef

Link to comment
Share on other sites

Link to post
Share on other sites

@Electronics Wizardy
" Parity raid really isn't that bad, and id probably do that here instead of mirrors. They normally rebuild at the max speed of the drive(or the same speed as a mirror here). "
I heard something about a not perfect chance of rebuilding rate with parity? Something about errors in this and this bit/byte? This made me instantly go "NOPE"

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, ZFSinmylungs said:

@Electronics Wizardy
" Parity raid really isn't that bad, and id probably do that here instead of mirrors. They normally rebuild at the max speed of the drive(or the same speed as a mirror here). "
I heard something about a not perfect chance of rebuilding rate with parity? Something about errors in this and this bit/byte? This made me instantly go "NOPE"

Yea there isn't a perfect chance of a rebuild, but this is also the same issue with mirrors. 

 

Id says mirrors are more likely to have a issue rebuilding than multiple parity as there is only one other copy of the data in a mirror, while parity in ZFS can have 3 other copies of the data.

 

ALso zfs is pretty good about skipping the small part of the error and rebuilding the rest of the data, and doing background scrubs.

 

 

Also backups, if something goes wrong, you can always just restore backup of the array. There is no perfect disk array.

Link to comment
Share on other sites

Link to post
Share on other sites

"Id says mirrors are more likely to have a issue rebuilding than multiple parity as there is only one other copy of the data in a mirror, while parity in ZFS can have 3 other copies of the data. "
Mirroring is just copying, isn't it?
And yeah.. is that right? I did hear something about "you can lose up to 3 disks until unrecoverable"? Also what if the rebuild fails? Can I just try again? lol I don't think so as things shouldn't be random.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, ZFSinmylungs said:

"Id says mirrors are more likely to have a issue rebuilding than multiple parity as there is only one other copy of the data in a mirror, while parity in ZFS can have 3 other copies of the data. "
Mirroring is just copying, isn't it?
And yeah.. is that right? I did hear something about "you can lose up to 3 disks until unrecoverable"? Also what if the rebuild fails? Can I just try again? lol I don't think so as things shouldn't be random.

Whats your backups plan? You should have backups so the whole array can fail and you still have all the data.

 

Mirroring is kinda like copying, but not exaclty the same. For a home server, id really suggest going raid z2 here. CHeaper per tb, better for your uses. Then spend the money on backups which are much better at protecting your data than raid/redundancy.

Link to comment
Share on other sites

Link to post
Share on other sites

@Electronics Wizardy
"Whats your backups plan? You should have backups so the whole array can fail and you still have all the data. "
Don't push it. The raid technology is meant to be an backup, you silly-nilly! No one has money to 10-times-backup their data -_-

"Mirroring is kinda like copying, but not exaclty the same. For a home server, id really suggest going raid z2 here."
I heard that's like RAID 5, hmmh? And yeah.. I was recommended this before o.o
What do you mean mirroring is not exactly like copying?

"CHeaper per tb, better for your uses. Then spend the money on backups which are much better at protecting your data than raid/redundancy."
Cheaper? Better? "Then spend moneys on backups" - "then" then what? You are talking about one thing then another thing.. change your mind, please? lol funny to read.
Mirroring should be in theory a backup.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, ZFSinmylungs said:

Mirroring should be in theory a backup.

Mirroring isn't a back, as there are a lot of ways that both of the mirrors can lose the data at the same time. Like if the filesystem got corrupted, the drives all died to a power surge, or the pc got ransomwared or stolen. Backups are much better at keeping your data safe.

 

1 hour ago, ZFSinmylungs said:

heard that's like RAID 5, hmmh? And yeah.. I was recommended this before o.o

Raid z2 is like raid 6, and what Id suggest you use here. You need less drives. Rebuild time will be similar. Spend the savings on a good backup plan instead of mirrors. You can never trust just one drive array.

 

1 hour ago, ZFSinmylungs said:

What do you mean mirroring is not exactly like copying?

Normally copies are independt copies, mirrors are managed by the same filesystem and os, so if something broke the filesystem, all your data would be gone.

Link to comment
Share on other sites

Link to post
Share on other sites

" Mirroring isn't a back, as there are a lot of ways that both of the mirrors can lose the data at the same time. Like if the filesystem got corrupted, the drives all died to a power surge, or the pc got ransomwared or stolen. Backups are much better at keeping your data safe. "
Sure I guess.
Filesystem corruption? Unless I mess with some GNU utilities or screw up my ZFS fs somehow.. I should be good.
All drives died due to a power surge? Hmm perhaps you mean like a lightning strike? If so - I would just entirely disconnect my computer.
Ransomwared or stolen? - Now you're pushing it a bit xD Like 99.99% of software I use is freedom software.. so I should be that.. unless you meant like a hack.. well.. I do use quite good security tools, but I haven't set up my firewall yet.. too damn lazy :d

"Raid z2 is like raid 6, and what Id suggest you use here. You need less drives. Rebuild time will be similar. Spend the savings on a good backup plan instead of mirrors. You can never trust just one drive array. "
So you're like recommending a offline backup or something?
if so: 1. that'd be very unconvenient 2. as expensive as raid 1/mirroring.

" Normally copies are independt copies, mirrors are managed by the same filesystem and os, so if something broke the filesystem, all your data would be gone. "
I guess you're right.. except.. like I said.. the filesystem.. I think I have heard ZFS is VERY well tried and proven. Except I'd also need ECC ram for much better reliability or so? It never failed me so far.

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, ZFSinmylungs said:

Filesystem corruption? Unless I mess with some GNU utilities or screw up my ZFS fs somehow.. I should be good.

Well ZFS can also just have issues. Also user error is a major issue.

 

Just now, ZFSinmylungs said:

All drives died due to a power surge? Hmm perhaps you mean like a lightning strike? If so - I would just entirely disconnect my computer.

Or the powersupply has a issue. There is always the change.

 

1 minute ago, ZFSinmylungs said:

Ransomwared or stolen? - Now you're pushing it a bit xD Like 99.99% of software I use is freedom software.. so I should be that.. unless you meant like a hack.. well.. I do use quite good security tools, but I haven't set up my firewall yet.. too damn lazy :d

Ive seen a good amount of linux boxes that get malware on them. Opensource doesn't mean secure.

 

1 minute ago, ZFSinmylungs said:

So you're like recommending a offline backup or something?

Yea a backup is really the only way to keep data safe here. Don't try to make the most reliable array, backup your array. There is a lot of things that can happen that can destroy all the data on the array.

 

But id still go raidz2 here. Get something like 6x 8tb drives for 32tb usable.

 

ALso Please quote people, its easy and notifies them.

Link to comment
Share on other sites

Link to post
Share on other sites

" Ive seen a good amount of linux boxes that get malware on them. Opensource doesn't mean secure. "
Open source does mean secureR. More coverage, more transparent.
Like what kind-of malware? That is interesting. Any idea by what way?

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

" But id still go raidz2 here. Get something like 6x 8tb drives for 32tb usable. "
Very interesting.
I am at the beginning before I really started investigating: I cannot decide lol. I was so sure of mirroring.

" ALso Please quote people, its easy and notifies them. "
The button for it disappears when not using JavaScript. I am and will not use JavaScript (on here.... anymore lol).

P.S. I am so fucking tired I made 2 comments again holy shit!!! I wanted to cut-paste into the edit of the comment before this one.... and without JS - the delete button is also not found..

Edited by ZFSinmylungs
Stupidity

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

5 minutes ago, ZFSinmylungs said:

" Ive seen a good amount of linux boxes that get malware on them. Opensource doesn't mean secure. "
Open source does mean secureR. More coverage, more transparent.
Like what kind-of malware? That is interesting. Any idea by what way?

Not really, yea it lets people check the code more, but there are still many large security issues in open source code(look at the cve list for linux for example).

 

ALso most security ive seen are due to configuration not software issues. Things like bad passwords, bad permissions, missing firewalls, old software versions. Open source won't help you here.

 

Ive seen boxes with crypto miners and ransomware. 

4 minutes ago, ZFSinmylungs said:

I am at the beginning before I really started investigating: I cannot decide lol. I was so sure of mirroring.

What are your exact needs of this nas? What network bandwidth?

 

For most home nas boxes raid z2 makes a lot more sense. 

 

Also really you need backups if you want to keep your data safe. Just put it on external hdds and keep them offsite. Raid/ZFS is not a backup and I have seen a lot of issues where the whole systems has its data lost and raid can't help you there.

Link to comment
Share on other sites

Link to post
Share on other sites

" ALso most security ive seen are due to configuration not software issues. Things like bad passwords, bad permissions, missing firewalls, old software versions. Open source won't help you here. "
I see I see, thanks!
bad passwords - check
bad permissions - should be checked
missing firewalls - fuck
old software versions - Arch
Open source - it is helping... again - 100% security? cannot exist. :/

" Ive seen boxes with crypto miners and ransomware. "
:d rip

" What are your exact needs of this nas? What network bandwidth?



For most home nas boxes raid z2 makes a lot more sense. "
Storage solution... long-term, with shit-load of room of upgrading I guess.. as I work I'd be adding more and more storage until one day I breakdown and fuuuuuuu burn down everything and go live in forest naked, with monkeys lol.
Soo raidz2 you say? Similar to raid 6 you say? I heard that raid 6 shouldn't even exist? Was it the Chris Titus Tech guy that I stopped watching long time ago? I suppose.

"Also really you need backups if you want to keep your data safe."
You are talking "safe" here and I suppose like super-ultra-high security like like datacentre-level? Because this is what it sounds like to me XD

"Raid/ZFS is not a backup and I have seen a lot of issues where the whole systems has its data lost and raid can't help you there. "
Again - you are still a stranger to me and since I've seen no issues YET.. it is quite hard to believe that "a lot of issues" part.. believe it or not.. but yeah. you did mention the user-error thing.. which I don't consider an issue.. a good administrator always reads the manual like 5 times..

I cannot quote people with the built-in function because I almost never enable JavaScript on websites. You won't get notified if I reply to you, sadly :/

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, ZFSinmylungs said:

bad passwords - check

ALso use 2f where you can and ssh certs instead of passwords where possible

 

2 minutes ago, ZFSinmylungs said:

Soo raidz2 you say? Similar to raid 6 you say? I heard that raid 6 shouldn't even exist? Was it the Chris Titus Tech guy that I stopped watching long time ago? I suppose.

Raid 6 is fine, IDK why people would say never use it. Id suggest raid z2 here. Probably go a bit overboard with storage so you have space to grow later on.

 

3 minutes ago, ZFSinmylungs said:

you did mention the user-error thing.. which I don't consider an issue..

User error is a massive issue. Ever admin makes mistakes, and many massive outages are due to user error. Don't assume your perect

 

4 minutes ago, ZFSinmylungs said:

You are talking "safe" here and I suppose like super-ultra-high security like like datacentre-level? Because this is what it sounds like to me XD

Well if you want to keep data, you really need backups. Raid is not a backup, and lots of things can cause your whole array to lose data. 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×