Noob questions about NAS, Servers and what they can do

[Electronics Wizardy]

2 hours ago, latot said:

A Syneology, what y read before, seems is good for NAS but not too good for anything else, I don't know if a torrent server will play fine with with, and if I want do other things..., maybe a Syneology with some Linux? what is the advatage/disadvantage betwen Syneology SO vs some Linux?

THey can run vms, so you can run programs you want on it. Synology systems also run linux under the hood.

 

2 hours ago, latot said:

Is hard have a raid with a DIY?

You can easily setup a raid with a diy nas, but there are lots of ways to do this.

 

2 hours ago, latot said:

 .. I dont' have where to start looking for with this...., I never thought with all the NAS dying...., yes..., is something that need to be considerated....

Cloud backup?

 

Or get anouther nas/external hdds to backup on to.

 

2 hours ago, latot said:

I was thinking in the partition system..., I don't like NTFS because the fragmentation, I usually use Ext4 in my system, but can be the best for a NAS.

Id decide on the os/nas model first. That will normally force the os you want to use.

[latot]

On 9/25/2021 at 4:55 PM, Electronics Wizardy said:

Id decide on the os/nas model first. That will normally force the os you want to use.

How is that, usually, linux run any place, in what case is better other SO?

[Electronics Wizardy]

1 minute ago, latot said:

How is that, usually, linux run any place, in what case is better other SO?

Well the linux distro makes a big difference, many of the premade distros are amde for a specific filesystem.

 

Im kinda a fan of ZFS, but it has its requirements.

[latot]

How can I start doing the DIY stuff?

 

Actually, I don't know how the bottlenecks works in a NAS, for example, with a CPU, how much MB/s can I have until the limit?, the SATA cable limit?, is there some motherboard bottleneck in transfer speed?, how can I know this thing to build a nas?, more than common sense, I would like to know how to know or identify this.

 

Maybe this question is too out of topic, you actually, already answer the topic, very very thx

[Electronics Wizardy]

33 minutes ago, latot said:

How can I start doing the DIY stuff?

 

Actually, I don't know how the bottlenecks works in a NAS, for example, with a CPU, how much MB/s can I have until the limit?, the SATA cable limit?, is there some motherboard bottleneck in transfer speed?, how can I know this thing to build a nas?, more than common sense, I would like to know how to know or identify this.

 

Maybe this question is too out of topic, you actually, already answer the topic, very very thx

Do you have any spare hardware you want to use in this nas?

 

Do you have a budget for a nas?

 

Id probably just use unraid for a os, easy to use + setup, and easy to expand later on.

 

Speed wise, its almost always network limited with gigabit, then disk limited, cpu isn't a issue.

 

How many drive bays do you waant?

[latot]

On 9/30/2021 at 4:23 PM, Electronics Wizardy said:

Do you have any spare hardware you want to use in this nas?

Sorry, too late but I'm here.

I have this old motherboards

Compat PC: SP#289767-001 P639604P1NR4MP AS#287579-101 REV 4

Asus OOO C4AB194-60-MBLE20-A01 1003

 

No idea even if can works for today.

 

On 9/30/2021 at 4:23 PM, Electronics Wizardy said:

Do you have a budget for a nas?

I'll have, not right now, but I like to do the things right, and considering efficiency, I don't need a super 1000TB of space, but lets consider the best part by part.

On 9/30/2021 at 4:23 PM, Electronics Wizardy said:

How many drive bays do you waant?

This is a hard question, without count the SO, I would like to have..., 30TB of space for now, maybe extend in the future, that would be 4 bays, 2 to store, and 2 for backup, I don't know if is a good idea, but have raid 0 with full cloud backup.

 

 

I was checking and reading more about VM for other projects, and I found something interesting, seems some shared folders works from the network adapter, so, even if I can "share" data with a project running in a vm inside the NAS, I will have a super botleneck.... this should be confirmed for example with synology, if don't have this problem the vm, that would be great.

[Electronics Wizardy]

3 minutes ago, latot said:

Sorry, too late but I'm here.

I have this old motherboards

Compat PC: SP#289767-001 P639604P1NR4MP AS#287579-101 REV 4

Asus OOO C4AB194-60-MBLE20-A01 1003

 

GOt links for those? Can't find it easily by googling

 

3 minutes ago, latot said:

This is a hard question, without count the SO, I would like to have..., 30TB of space for now, maybe extend in the future, that would be 4 bays, 2 to store, and 2 for backup, I don't know if is a good idea, but have raid 0 with full cloud backup.

Id probalby run raid 5 or 6 here, just keeps risk down.

 

You don't want backup in the same system, Id store backups on anouther system or with external hdds

 

4 minutes ago, latot said:

 

I was checking and reading more about VM for other projects, and I found something interesting, seems some shared folders works from the network adapter, so, even if I can "share" data with a project running in a vm inside the NAS, I will have a super botleneck.... this should be confirmed for example with synology, if don't have this problem the vm, that would be great.

What do you mean exactly? With a network share the vm won't matter. Do you want to do things on this nas other than data storage?

 

 

[latot]

3 minutes ago, Electronics Wizardy said:

GOt links for those? Can't find it easily by googling

Me too, sadly.

 

11 minutes ago, Electronics Wizardy said:

What do you mean exactly? With a network share the vm won't matter. Do you want to do things on this nas other than data storage?

Usually, I have two types of multimedia files, one of them is the ones that is "work in progress", but the other ones work as backup of previous works and some sources, to keep everything right I have this:

 

1) to share, use torrent, every .torrent file have the data to verify if everithing is ok, I have a python script that loads a custom file for every .torrent file, and, it helps me to sort all the data without the usual torrent programs, where all is in the same folder. But share and verify need speed, or with a lot of data will take forever if there is a problem, or if is sharing, if the vm uses the network adapter, the data transfer will be limited to it, and is not too fast to share data/verify data/and work with the data in a local connection.....

 

2) As every multimedia project, there is several source files, and when for example, merging a new file, depending of how we do it, there can be a lot of repeated data, so, to solve this, I'm working in a custom filesystem, basically, this filesystem, read data from other sections of files that I specify previously, and I can save a lot of space, this filesystem need to works over the NAS, and from there expose other things for example, to the python script, so in that way, even if there is some slows parts, I can share several files saving a lot of space.

[Electronics Wizardy]

27 minutes ago, latot said:

Me too, sadly.

What CPU does it take? What socket?

 

28 minutes ago, latot said:

 

2) As every multimedia project, there is several source files, and when for example, merging a new file, depending of how we do it, there can be a lot of repeated data, so, to solve this, I'm working in a custom filesystem, basically, this filesystem, read data from other sections of files that I specify previously, and I can save a lot of space, this filesystem need to works over the NAS, and from there expose other things for example, to the python script, so in that way, even if there is some slows parts, I can share several files saving a lot of space.

You know they already have dedup in lots of filesystems? Id just use a tried and true dedup system. You don't want the filesystem to have issues, and a good filesystem is a lot of work.

 

28 minutes ago, latot said:

Usually, I have two types of multimedia files, one of them is the ones that is "work in progress", but the other ones work as backup of previous works and some sources, to keep everything right I have this:

 

Id just get one big raid 5/6 here. Will be plenty of speed for checksums.

 

Id probably go something like 6x12tb hdds in raid 6. Gives you space to grow if you want too.

[latot]

21 minutes ago, Electronics Wizardy said:

What CPU does it take? What socket?

Uff, I don't have right now the motherboards here..., there are in a place that is hard to get now.

 

22 minutes ago, Electronics Wizardy said:

You know they already have dedup in lots of filesystems? Id just use a tried and true dedup system. You don't want the filesystem to have issues, and a good filesystem is a lot of work.

Actually, the the idea of the project is to works over Windows/Linux, a custom filesystem over the library FUSE do the trick, and is safe, I want to use this in other things too, so don't worry, but is true if I do a mistake I can corrupt my data...., in that case, I can do the project to work over read-only.

 

I knew there is filesystems with dedupe, but there is a lot of "costs" in speed and a lot of things about it, and if I know what project uses what files, I can do that works once, instead the fs doing it all the time, even if I want I can disable it, or access directly to the files. The fs ZFS lack of support, so I would like to avoid it.

[Electronics Wizardy]

4 hours ago, latot said:

 

I knew there is filesystems with dedupe, but there is a lot of "costs" in speed and a lot of things about it, and if I know what project uses what files, I can do that works once, instead the fs doing it all the time, even if I want I can disable it, or access directly to the files. The fs ZFS lack of support, so I would like to avoid it.

ZFS with a special device for dedup is pretty reasonble. 

 

There is also offlne dedup soltuions that work pretty well, and don't have the big performance hit. BTRFS can do this, along with ntfs/refs. 

 

Id avoid doing a diy filesystem, esp as dedup has been done by many others already.

 

 

[latot]

Hi, is that complicated, at least a custom filesystem with FUSE is different as a usual filesystem, summary, we can run windows/linux in any filesystem, lets say we have ntfs, then inside we can "mount" our filesystem, like a virtual filesystem. This type of custom filesystem don't need to installed in any place like a normal filesystem, just consider it an "app" that mount a "folder" that is readed and wrote in a custom way. This is not unusual, is just this type of things are hidden in software. A big advatage mixing this, is that I can copy data from one fs to other, without get more size, so is very handy. Note, made a new fs like ntfs/ext4 to use it as a base filesystem (installed in the partition) I agree can be a very bad idea.

 

Oks, returning to the main point.

 

Use a VM for something like this, or share torrent seems a very bad idea, I'll contact synology to confirm how the shared data works in their vm.

 

For a custom build..., we talk to get a motherboard, in the best case with the new ethernet.

 

If the CPU don't cause any bootleneck, why some NAS use a powerful CPU?

[Electronics Wizardy]

1 hour ago, latot said:

 

Use a VM for something like this, or share torrent seems a very bad idea, I'll contact synology to confirm how the shared data works in their vm.

What do you mean by shared data? 

 

A vm for a torrent system will work fine.

 

1 hour ago, latot said:

For a custom build..., we talk to get a motherboard, in the best case with the new ethernet.

 

What ethernet standards are ou using? There are dozens of them. Are you just using normal 1000base-t?

 

1 hour ago, latot said:

 

 

If the CPU don't cause any bootleneck, why some NAS use a powerful CPU?

Cause they want to do things other than just sharing files, like transcoding meda. For sharing files you don't need much at all.

 

 

[latot]

3 hours ago, Electronics Wizardy said:

What do you mean by shared data? 

 

A vm for a torrent system will work fine.

As we talk before, actually there is some functions that I need:

 

Share from torrent

Access to the data locally

 

If I run the torrent from VM, and the VM use the network adaper, use both, torrent and local work, can collapse the network, or just say can work as a bottleneck.

 

4 hours ago, Electronics Wizardy said:

What ethernet standards are ou using? There are dozens of them. Are you just using normal 1000base-t?

Yes, the normal one, but I can change it if needed, and probably needed.

 

4 hours ago, Electronics Wizardy said:

Cause they want to do things other than just sharing files, like transcoding meda. For sharing files you don't need much at all.

Thx, I'm not interested in transcode from the NAS.

[Electronics Wizardy]

1 minute ago, latot said:

f I run the torrent from VM, and the VM use the network adaper, use both, torrent and local work, can collapse the network, or just say can work as a bottleneck.

 

the vm is bridged by default, and will work like anouther pc on the network.

 

1 minute ago, latot said:

Access to the data locally

Basically any nas can do this, via smb normally.

 

Yea any nas should meet your needs. Just add a bit of ram for a vm. Id get something like a DS920+ and put something like 4x 12-16tb hdds in it in a raid 5 and your good.

[latot]

2 hours ago, Electronics Wizardy said:

the vm is bridged by default, and will work like anouther pc on the network.

mmm, I see..., we can skip some problems, if the hdd is exposed from the VM...., but this seems maybe a bad idea?, feels like, get a Synology, make a VM with linux to mount and share data, then expose the folder from the VM to access to locally..., and how to do that, would be from an app inside the VM, not from the NAS directly...

 

2 hours ago, Electronics Wizardy said:

Basically any nas can do this, via smb normally.

 

Yea any nas should meet your needs. Just add a bit of ram for a vm. Id get something like a DS920+ and put something like 4x 12-16tb hdds in it in a raid 5 and your good.

Seems a good option.

[Electronics Wizardy]

2 minutes ago, latot said:

mmm, I see..., we can skip some problems, if the hdd is exposed from the VM...., but this seems maybe a bad idea?, feels like, get a Synology, make a VM with linux to mount and share data, then expose the folder from the VM to access to locally..., and how to do that, would be from an app inside the VM, not from the NAS directly...

Think of the vm like any other system on your network. It can access the shares on the nas just like any other box. It doesn't have direct access to the drives on the nas.

[latot]

Yes, I'm thinking this  comparison, actually share from the vm, would be the same as a custom build of a NAS, even a build would be better because there a direct connection to the hdds..., from here, seems a Synology is not very effective.

 

For hadware, Synology seems great.

[Electronics Wizardy]

2 hours ago, latot said:

Yes, I'm thinking this  comparison, actually share from the vm, would be the same as a custom build of a NAS, even a build would be better because there a direct connection to the hdds..., from here, seems a Synology is not very effective.

 

A diy nas would work exactly the same here. 

 

You can run a torrent app on the synology directly too, you don't need a vm, Id just use a vm as you have a failry custom config

 

Also there doing it over the network won't hurt you here, its basically the same speed as the local disk.

[latot]

What would be the advatange of Synology over a custom NAS in this case?

 

And, I thought about the next two things.

 

1) Here, I have from time to time ,heartquakes...., and that with hdd is a very bad idea, exist something to cover that?

2) Here is..., low power some times, or energy cuts..., that to a NAS seems a very bad idea too, I thought in a UPS, but I don't know if the NAS can detect in some way the energy cut to shutdown.

[Electronics Wizardy]

1 hour ago, latot said:

What would be the advatange of Synology over a custom NAS in this case?

 

Nice hardware, support, no need to setup the os and configure it. 

 

1 hour ago, latot said:

1) Here, I have from time to time ,heartquakes...., and that with hdd is a very bad idea, exist something to cover that?

 

I woudln't worry about earth quakes, small ones won't break anything, and big ones are very rare, and thats why you have backups.

 

1 hour ago, latot said:

2) Here is..., low power some times, or energy cuts..., that to a NAS seems a very bad idea too, I thought in a UPS, but I don't know if the NAS can detect in some way the energy cut to shutdown.

Almost all nas units can talk to a supported nas to shut down automatically.

[latot]

10 hours ago, Electronics Wizardy said:

Nice hardware, support, no need to setup the os and configure it.

If I mount a VM to share, I'll need configure it, but the raid system can be handled from Synology, can this raid system be mounted without the Synology OS in other linux?

 

10 hours ago, Electronics Wizardy said:

I woudln't worry about earth quakes, small ones won't break anything, and big ones are very rare, and thats why you have backups.

Here, are not unusual, log earth quakes are weird, but when it comes are pretty intense, for the short ones, can be very intense too..., in richter termns, the small ones are from 4 to 5, the big ones..., over 7 to 9...., for example today, have already been 4 earthquakes, from 3.1 to 4.9...., well nothing happens really, all constructions are mode to resist, even a big one don't cause much trouble, but for hardware...., due to climate + earthquakes we have power problems and hardware problems. So is very important cover earthquakes :3

[Electronics Wizardy]

31 minutes ago, latot said:

f I mount a VM to share, I'll need configure it, but the raid system can be handled from Synology, can this raid system be mounted without the Synology OS in other linux?

Yea its normal linux md raid + lvm, any linux distro should work fine.

 

31 minutes ago, latot said:

11 hours ago, Electronics Wizardy said:

 

Here, are not unusual, log earth quakes are weird, but when it comes are pretty intense, for the short ones, can be very intense too..., in richter termns, the small ones are from 4 to 5, the big ones..., over 7 to 9...., for example today, have already been 4 earthquakes, from 3.1 to 4.9...., well nothing happens really, all constructions are mode to resist, even a big one don't cause much trouble, but for hardware...., due to climate + earthquakes we have power problems and hardware problems. So is very important cover earthquakes :3

Yea those small ones won't cause an issue, and for the big ones restore backups. Make sure you have a backup of all the data you care about. 

[latot]

4 hours ago, Electronics Wizardy said:

Yea those small ones won't cause an issue, and for the big ones restore backups. Make sure you have a backup of all the data you care about. 

Yes, is good idea have a backup, and I would like a way to not loose the NAS with the earthquakes....

[Electronics Wizardy]

25 minutes ago, latot said:

Yes, is good idea have a backup, and I would like a way to not loose the NAS with the earthquakes....

Well ssds would be the best option here, but Id sy not worth the cost.

 

Mounting the nas well so it won't fall will help. Id put it on/near the floor or in a space it can't fall from. Or put it in a rack that is screwed down

 

And really, earth quakes aren't that big of a issue. Pretty rare and small ones won't cause problems, only the big ones are likely to cause issues, and you will probably have more issues than the nas not working.