Synology hard limits some NAS systems to 16 TB Pool size do to "critical overflow issue"

[jagdtigger]

https://www.reddit.com/r/synology/comments/gqv61z/storage_pool_has_exceeded_this_models_16_tb/frvwo7d/

Quote

spoke with support regarding this and here's their response:

"This issue has to do with the 32-bit CPU used in older models. Basically, these models could only handle up to 232 addresses, which is where the 16 TB limit comes from: 232 times 4,096 bytes = 16 TB.

The issue is that Synology recently discovered a critical overflow issue that can cause data corruption if the total size (not used size) of volumes in a storage pool exceeds 16 TB in a 32-bit NAS. Thus, we suggest that you keep the volume at or below 16 TB and don't migrate your storage pool to enable multiple volume support. If your volume is less than or equal to 16 TB you can keep using the volume, but if you have a storage pool with volumes totalling more than 16 TB we strongly recommend backing up your data and removing the excess volumes.

 

My take on this:

I have a DS416 for 3 years with 4 10 TB drives in raid for 3 years  now without issues. Yet they still decided to introduce a hard limit after the fact. To top it off it is not even mentioned in the change log, no detailed info about the issue anywhere on their site either. This smells like to me as if they are trying to force ppl to upgrade even though their NAS runs fine...

[Master Disaster]

2 minutes ago, jagdtigger said:

https://www.reddit.com/r/synology/comments/gqv61z/storage_pool_has_exceeded_this_models_16_tb/frvwo7d/

 

My take on this:

I have a DS416 for 3 years with 4 10 TB drives in raid for 3 years  now without issues. Yet they still decided to introduce a hard limit after the fact. To top it off it is not even mentioned in the change log, no detailed info about the issue anywhere on their site either. This smells like to me as if they are trying to force ppl to upgrade even though their NAS runs fine...

It sounds like this will only become an issue if the CPU has to address data in a range over the 16TB addresses. In other words once you use more than 16TB on your pool it becomes an issue, before that it will probably be fine. That might explain why you haven't encountered it yet.

[3rrant]

4 minutes ago, comander said:

One of the reasons I bought a NAS for a relative's small business was the have fewer IT related issues. 

 

While this issue doesn't affect the install I'm referencing, it does make me raise an eye brow

You'd still have (probably) more issues if you went with a complete custom solution than relaying on another's company but they aren't immune to everything either

[jagdtigger]

16 minutes ago, Master Disaster said:

It sounds like this will only become an issue if the CPU has to address data in a range over the 16TB addresses. In other words once you use more than 16TB on your pool it becomes an issue, before that it will probably be fine. That might explain why you haven't encountered it yet.

It is possible. But i havent found any info on limits regarding raid pool size on a 32 bit system, and this bothers me. And they didnt even disclose it, its fishy.....

[leadeater]

18 minutes ago, jagdtigger said:

It is possible. But i havent found any info on limits regarding raid pool size on a 32 bit system, and this bothers me. And they didnt even disclose it, its fishy.....

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/Storage/Why_does_my_Synology_NAS_have_a_single_volume_size_limitation

 

You can have more than 16TB storage in the affected models, you just can no longer create a single volume larger than 16TB. Being 32bit hardware it's likely a very real problem.

 

Also the limits are tied to what created the volume in the first place so it sounds like different models have slightly different RAID implementations or write different metadata information on to the array.

Quote

Before you do so, please note that the maximum single volume size is determined when the volume is created. Therefore, migrating the drives from one Synology NAS supporting 16 TB to another Synology NAS supporting 108 TB does not mean the maximum single volume size will expand from 16 TB to 108 TB. Instead, only the new volumes created in the same storage pool can support up to 108 TB.

 

[leadeater]

4 minutes ago, Escanor said:

Guess my ds218play is fine then ?

108TB limit for yours

[Sauron]

Even more reasons to avoid dedicated NAS units in favor of a low power PC if at all possible. The cost is similar (sometimes lower for a PC) and you get much more flexibility to do whatever you want with it without depending on the whims and unwarranted cost saving measures of the manufacturer.

[Windows9]

4 minutes ago, Sauron said:

Even more reasons to avoid dedicated NAS units in favor of a low power PC if at all possible.

I'm using a banana pi attached to a solar panel

[jagdtigger]

11 minutes ago, leadeater said:

You can have more than 16TB storage in the affected models, you just can no longer create a single volume larger than 16TB. Being 32bit hardware it's likely a very real problem.

That was in place for a long time, what they limited now is the pool size.

 

[LogicalDrm]

-> Moved to Servers and NAS

 

This doesn't seem like news.

[Master Disaster]

35 minutes ago, Sauron said:

Even more reasons to avoid dedicated NAS units in favor of a low power PC if at all possible. The cost is similar (sometimes lower for a PC) and you get much more flexibility to do whatever you want with it without depending on the whims and unwarranted cost saving measures of the manufacturer.

And you pretty much have to learn Linux plus all the intricacies of managing storage devices over a network. I'm incredibly familiar with Linux, understand the basics of Pools, Volumes, RAID, Samba etc and I still chose a Synology over a custom system simply because 90% of the work was done for me OOTB. It's not that I couldn't have done it myself (before my DSM I was running a Pi 4 with a USB HDD and OMV where I had to do everything manually), it was that I knew it was just going to work all the time with out the need for me to be constantly in SSH running repair and maintenance tasks.

 

Its a trade off and I got so sick of fighting with OMV that I chose the easy option.

[LAwLz]

4 minutes ago, Master Disaster said:

And you pretty much have to learn Linux plus all the intricacies of managing storage devices over a network. I'm incredibly familiar with Linux, understand the basics of Pools, Volumes, RAID, Samba etc and I still chose a Synology over a custom system simply because 90% of the work was done for me OOTB. It's not that I couldn't have done it myself (before my DSM I was running a Pi 4 with a USB HDD and OMV where I had to do everything manually), it was that I knew it was just going to work all the time with out the need for me to be constantly in SSH running repair and maintenance tasks.

 

Its a trade off and I got so sick of fighting with OMV that I chose the easy option.

And with something like a Synology you get a very "slick" system as well. Rather than a large tower with loud fans that sucks a lot of power, you get a box with a really small footprint designed for NAS usage in mind. Hot swap without having to open the chassi. Drive indicators for each individual drive. A single (or maybe two) fans to cool everything in an efficient way. Stuff like that adds value.

 

I say that as a DS1817+ owner who considered building a computer for NAS usage instead. It ended up being roughly the same price but with the drawbacks I just mentioned. Of course I sacrificed some things too like customization and the ability to possibly run other software on it, but that was a sacrifice I was willing to make.

[elfensky]

Isn't this more a convenience issue rather than a usability issue?

Correct me if I'm wrong but nothing should stop you from being able to create multiple 16TB volumes, so you should still be able to use "all of the space", it just won't be as convenient as having 1 huge pool

And if you partition your drives correctly you should be able to still get the full benefits of raid.
Lets say you have 4x8TB drive, you partition each drive to have 2 4TB partitions.

You then just make 2 16TB volumes, with:
A1, B1, C1, D1 and
A2, B2, C2, D2

the volume is still split across 4 drives with redundancy. It should even still be hot swappable that way.

[Sauron]

8 minutes ago, Master Disaster said:

And you pretty much have to learn Linux plus all the intricacies of managing storage devices over a network. I'm incredibly familiar with Linux, understand the basics of Pools, Volumes, RAID, Samba etc and I still chose a Synology over a custom system simply because 90% of the work was done for me OOTB. It's not that I couldn't have done it myself (before my DSM I was running a Pi 4 with a USB HDD and OMV where I had to do everything manually), it was that I knew it was just going to work all the time with out the need for me to be constantly in SSH running repair and maintenance tasks.

 

Its a trade off and I got so sick of fighting with OMV that I chose the easy option.

Or you can use freenas, or unraid, or any other system designed to make this procedure as simple as possible. It'll never be quite as plug and play as an off the shelf NAS but I'd say it gets close enough that the flexibility you get in return is well worth it.

1 minute ago, LAwLz said:

Hot swap without having to open the chassi.

This is a thing on a variety of affordable small form factor systems. I use an old Dell Microserver for this and it is every bit as serviceable as a dedicated NAS.

[Sauron]

4 minutes ago, elfensky said:

Isn't this more a convenience issue rather than a usability issue?
Nothing should stop you from creating multiple 16TB volumes, so you should still be able to use "all of the space", it just won't be as convenient as having 1 huge pool, instead being split over multiple volumes.

And if you partition your drives correctly you should be able to still get the full benefits of raid.
Lets say you have 4x8TB drive, you partition each drive to have 2 4TB partitions.

You then just make 2 16TB volumes, with:
A1, B1, C1, D1 and
A2, B2, C2, D2

the volume is still split across 4 drives with redundancy

That's still a pretty low limit if you're using 6, 8, 10 or 12TB drives and especially if you want a RAID 5 or 6 setup with more than 2 drives.

[jagdtigger]

24 minutes ago, elfensky said:

Isn't this more a convenience issue rather than a usability issue?

There are two issues, A: this is after the fact so now even a perfectly ok system is full of warnings and disabled data scrubbing(<- big no-no), B: there is no credible information on the internet that a 32 bit system can corrupt a 16TB+ array, if that were the case mdadm wouldnt even let you create one on it and/or mount it on a 32 bit system like ext4 does.....

[leadeater]

20 hours ago, jagdtigger said:

There are two issues, A: this is after the fact so now even a perfectly ok system is full of warnings and disabled data scrubbing(<- big no-no), B: there is no credible information on the internet that a 32 bit system can corrupt a 16TB+ array, if that were the case mdadm wouldnt even let you create one on it and/or mount it on a 32 bit system like ext4 does.....

mdadm isn't Synology's RAID implementation, just because one doesn't have the issue doesn't mean the other does not and neither does it mean it can be fixed either. Sure you could fix it with a re-write/re-implementation of RAID but that would mean your current array wouldn't support the fix and you'd have to backup and erase the entire thing.

 

So unless Synology ground up changes the way they do RAID no fix is coming, not when it works fine on 64bit hardware and they can just stop using 32bit on NAS systems with more than 2 bays, or all together if they choose.

 

Any change like this would have come from a customer support ticket and an engineer investigation in to the issue and this is the root cause of the support case. Synology the company has to think about everyone and what might happen to them, if this could happen to anyone regardless of it hasn't happened yet then they have to release an update to fix the issue, if that means removing support for 16TB+ pools then that just has to happen.

 

It's going to look a lot worse if Synology knew about a rare but potential data corruption issue that gets more likely as the pool size gets bigger and does nothing about it then customers start swapping in 10TB, 12TB, 14TB & 16TB disks then data corruption reports spike like crazy. "We thought the chances of data corruption were too rare so we did nothing about it", imagine how well that would go down on reddit.

 

Bottom line mdam is not Synology. It's been running fine is the famous last words of everyone who then gets said problem.

[jagdtigger]

16 minutes ago, leadeater said:

mdadm isn't Synology's RAID implementation, just because one doesn't have the issue doesn't mean the other does not

They are using mdadm:

Spoiler

ash-4.3# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[4]
      29284844544 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3]
      2097088 blocks [4/4] [UUUU]
      
md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3]
      2490176 blocks [4/4] [UUUU]
      
unused devices: <none>
ash-4.3# mdadm
Usage: mdadm --help
  for help

 

And the issue is the other way around, there are 0 cases on the net where anything happened that is  similar to what they claim. And to disable data scrubbing is going way too far IMO. Now storage mgmt is full of warnings so i wont be seeing if there is a real issue so thanks synology. You will get ditched after my exams end.

[leadeater]

11 minutes ago, jagdtigger said:

They are using mdadm:

Last time I used Synology they had traditional and also their own, SHR-1 & SHR-2.

 

11 minutes ago, jagdtigger said:

And the issue is the other way around, there are 0 cases on the net where anything happened that is  similar to what they claim

Well I have never posted on reddit or twitter the issues I've had with Netgear NAS's and support escalations to their engineers and the custom firmware they have written to fix the reported issue. Not everyone runs and posts issues online you know, especially businesses. 

[jagdtigger]

1 minute ago, leadeater said:

Well I have never posted on reddit or twitter the issues I've had with Netgear NAS's and support escalations to their engineers and the custom firmware they have written to fix the report issue. Not everyone runs and posts issues online you know, especially businesses. 

Since they using standard tools if they have issues with one others will too. And i think both of us know how ppl like to reuse old equipment , meaning there are a lot of 32 bit systems out there running mdadm.......

 

4 minutes ago, leadeater said:

Last time I used Synology they had traditional (mdadm) and also their own, SHR-1 & SHR-2.

I avoid vendor specific solutions generally. At least if the NAS fails i can just hook it up to a PC and get the data that way with mdadm.

[leadeater]

3 minutes ago, jagdtigger said:

Since they using standard tools if they have issues with one others will too. And i think both of us know how ppl like to reuse old equipment , meaning there are a lot of 32 bit systems out there running mdadm.......

Yes but that doesn't mean the issues is with Synology systems using mdadm, no point asking me why they took something away if it only affects 2 of the ~7 RAID choices as I wouldn't know why. Plus you/I don't even know if the issues is somewhere else in the software and it has nothing to do with the RAID implementation and it's some other data management software in the NAS that corrupts the data.

 

Edit:

There is literally no reason to make this change without a legitimate reason, Synology right now have brand new models with 32bit CPUs in them, this isn't some force upgrade thing for old/legacy systems/users.

[jagdtigger]

2 minutes ago, leadeater said:

Plus you/I don't even know if the issues is somewhere else in the software and it has nothing to do with the RAID implementation and it's some other data management software in the NAS that corrupts the data.

That part is their job to fix then . Its a different matter that they wont because its not good for their bottom line.

[leadeater]

1 minute ago, jagdtigger said:

That part is their job to fix then . Its a different matter that they wont because its not good for their bottom line.

You have the fix, disabled support for larger pools on 32bit hardware and it's likely all future models will not use 32bit CPUs anymore. And if you go look at my edit you'll see that is not even relevant since they sell 32bit hardware right now. There simply may be no possible way to fix the issue on 32bit hardware without requiring a complete wipe of the data and re-setup which is just bad user experience.

[Master Disaster]

1 minute ago, leadeater said:

You have the fix, disabled support for larger pools on 32bit hardware and it's likely all future models will not use 32bit CPUs anymore. And if you go look at my edit you'll see that is not even relevant since they sell 32bit hardware right now. There simply may be no possible way to fix the issue on 32bit hardware without requiring a complete wipe of the data and re-setup which is just bad user experience.

If its a hardware addressing issue there may be no way to fix it at all. If the CPU cannot count above x value then that's a hardware issue.

[jagdtigger]

On 5/28/2020 at 9:17 AM, Master Disaster said:

If its a hardware addressing issue there may be no way to fix it at all. If the CPU cannot count above x value then that's a hardware issue.

Judging how it never had issues with data scrubbing during the 3 years and the recent rebuild (hdd fail) i dont think its HW related. Anyway after my exam period ends it will be reassigned to hold my scratch discs. I already moved my home server to xcp-ng so all i need is something to hold the HDD's and a SATA HBA.