Whonnock stuff...

[BenzyVenue]

Helloooo,

I'm a noob with servers so pls don't roast me lol.

 

I was wondering why does Whonnock 3 need so much RAM and CPU cores?

If all the editors are reading/writing whatever information to Whonnock's M.2s, and Linus said he isn't storing any cache on the RAM, what's the purpose of all that hardware?

How is it possible that the CPU's can bottleneck on a file server, since whenever the editors load a Premiere Pro project, it just reads the information off of the M.2s, and brings it to the computer's RAM, right? And whenever a project would be saved, it would just get read off of the computer's RAM and write back onto Whonnock. I might be missing something here...

 

I'm a bit lost and curious about all this stuff, so any input would help!

Thanks!

 

[Exsosus]

You won't be judged here @BenzyVenue, that's what tech forums are for, to help individuals! Can I just ask you one direct question to begin this reply thread? If you are newbie to servers in general, why are you asking a sophisticated Whonnock 3 server question? Are you being humble? 

I just realized what you meant... you commenting Linus' new Whonnock 3 video... got it!

[BenzyVenue]

@Exsosus

 

Kinda... heh.

 

I guess I just don't really want to show off as knowing something, and then saying something stupid like I may have just said lol.

[LIGISTX]

6 minutes ago, BenzyVenue said:

Helloooo,

I'm a noob with servers so pls don't roast me lol.

 

I was wondering why does Whonnock 3 need so much RAM and CPU cores?

If all the editors are reading/writing whatever information to Whonnock's M.2s, and Linus said he isn't storing any cache on the RAM, what's the purpose of all that hardware?

How is it possible that the CPU's can bottleneck on a file server, since whenever the editors load a Premiere Pro project, it just reads the information off of the M.2s, and brings it to the computer's RAM, right? And whenever a project would be saved, it would just get read off of the computer's RAM and write back onto Whonnock. I might be missing something here...

 

I'm a bit lost and curious about all this stuff, so any input would help!

Thanks!

 

ZFS needs a lot of RAM to perform optimally, the rule of thumb is 1GB of RAM per TB of array space. But, more RAM = more better. And when your trying to provide massive bandwidth, more is always better. 
 

The reason for so much CPU requirement is similar. Normally a CPU more or less waits around for a harddrive or SSD to “gather” the data and then the CPU does what it does with said data. But on a nvme SSD array with the types of bandwidth they need, the CPU is actually being slammed with data; iirc they had to edit how this file handling is done so instead of waiting the CPU would actively seek the data from the array. Basically, with the amount of bandwidth the nvme array can provide, everything becomes almost a “bottleneck” just trying to get the data out the network interface. The nvme array can provide more bandwidth then any standard desktop’s RAM can…. Turns out OS’s were not really designed for this, yet

[RONOTHAN##]

10 minutes ago, BenzyVenue said:

How is it possible that the CPU's can bottleneck on a file server,

If you're throwing enough data around fast enough, then yes. Look at the Whonnock 2 saga, where they had half the CPU cores, the ran into CPU bottlenecks a lot faster (in addition to drive issues). 

 

12 minutes ago, BenzyVenue said:

and brings it to the computer's RAM, right?

No, actually. Because the storage in the system is so fast, it and RAM are actually about the same speed. The RAM is currently setup to hold only metadata. Think of it as a giant DRAM cache, more or less.

 

What you're thinking of is the way ZFS traditionally works. ZFS is designed around hard drives, so for that workload, the method you described works pretty well. For solid state, memory speed quickly becomes a bottleneck. 

[Kilrah]

-> Moved to Servers and NAS.

[BenzyVenue]

4 minutes ago, LIGISTX said:

ZFS needs a lot of RAM to perform optimally, the rule of thumb is 1GB of RAM per TB of array space. But, more RAM = more better. And when your trying to provide massive bandwidth, more is always better. 
 

The reason for so much CPU requirement is similar. Normally a CPU more or less waits around for a harddrive or SSD to “gather” the data and then the CPU does what it does with said data. But on a nvme SSD array with the types of bandwidth they need, the CPU is actually being slammed with data; iirc they had to edit how this file handling is done so instead of waiting the CPU would actively seek the data from the array. Basically, with the amount of bandwidth the nvme array can provide, everything becomes almost a “bottleneck” just trying to get the data out the network interface. The nvme array can provide more bandwidth then any standard desktop’s RAM can…. Turns out OS’s were not really designed for this, yet

Ahh. I understand. Thank you!

 

BTW, the reason I asked this in the first place was since I am helping a friend start up a business of about 20ish people, on a 10gig network. (We have all the networking switches and whatnot ready to go). This is an architectural company, so each Revit file is about 0.5 gb. So I am planning on making a server with M.2s which the employees work directly off of. I will have 8 2tb M.2s, for a total of 16tb, running in probably Raid 6. So, based on what you said, I am going to just use regular Desktop RAM, probably 32gb of it. And since its such small files, and less people constantly writing/reading onto it, I wouldn't really need a crazy CPU, so something like a standard i7 would do.

[BenzyVenue]

7 minutes ago, RONOTHAN## said:

If you're throwing enough data around fast enough, then yes. Look at the Whonnock 2 saga, where they had half the CPU cores, the ran into CPU bottlenecks a lot faster (in addition to drive issues). 

 

No, actually. Because the storage in the system is so fast, it and RAM are actually about the same speed. The RAM is currently setup to hold only metadata. Think of it as a giant DRAM cache, more or less.

 

What you're thinking of is the way ZFS traditionally works. ZFS is designed around hard drives, so for that workload, the method you described works pretty well. For solid state, memory speed quickly becomes a bottleneck. 

I see! I'm starting to get this lol. Thanks.

[LIGISTX]

2 minutes ago, BenzyVenue said:

Ahh. I understand. Thank you!

 

BTW, the reason I asked this in the first place was since I am helping a friend start up a business of about 20ish people, on a 10gig network. (We have all the networking switches and whatnot ready to go). This is an architectural company, so each Revit file is about 0.5 gb. So I am planning on making a server with M.2s which the employees work directly off of. I will have 8 2tb M.2s, for a total of 16tb, running in probably Raid 6. So, based on what you said, I am going to just use regular Desktop RAM, probably 32gb of it. And since its such small files, and less people constantly writing/reading onto it, I wouldn't really need a crazy CPU, so something like a standard i7 would do.

Depending on the needs…. I wouldn’t go i7 as it doesn’t support ECC ram. I would think a company of that size would easily spend the extra money in a Xeon platform, server hardware, and especially ECC RAM. Also……. I think you will find some issues with RAID and nvme. You have to also remember, a single nvme drive can typically do 2500 MB/s read, slow ones do 2500, fast ones do 5000+ easily. That’s already faster then your 10GBe network interface. RAIDing multiple together may just lead to exciting issues. I would look into ZFS and truenas specifically, and before you purchase hardware I would go to the truenas forums and post the full system requirements; they will be able to help you correctly spec and configure this. I know enough to know I don’t know enough to fully help you, but I know if your looking to deploy a nvme ZFS array you will need to really do your homework. And if you don’t use ZFS…. I would argue your doing it wrong especially for this application. ZFS snapshot alone makes it worth while for a business. 

[RONOTHAN##]

4 minutes ago, BenzyVenue said:

Ahh. I understand. Thank you!

 

BTW, the reason I asked this in the first place was since I am helping a friend start up a business of about 20ish people, on a 10gig network. (We have all the networking switches and whatnot ready to go). This is an architectural company, so each Revit file is about 0.5 gb. So I am planning on making a server with M.2s which the employees work directly off of. I will have 8 2tb M.2s, for a total of 16tb, running in probably Raid 6. So, based on what you said, I am going to just use regular Desktop RAM, probably 32gb of it. And since its such small files, and less people constantly writing/reading onto it, I wouldn't really need a crazy CPU, so something like a standard i7 would do.

I'm just gonna say, unless you really know what you're doing, you should probably hire a professional. 

 

Also, 10G networking is able to be saturated by HDDs, without too much effort as well. Just make sure to get a lot of RAM for the system. Half the reason LTT went all M.2 is because they really like overkill, and they're also using 40Gb networking to everyone, with dual 100Gb going to the server. You're no where near those speeds on a 10gig network. 

[LIGISTX]

Oh. I read m.2 as nvme. Will they be nvme or SATA m.2? If sata, you won’t have the extent of issues I mentioned above.

 

But either way, ECC should be used, and you will likely want 64GB of RAM at a minimum. But again, I’d look at truenas forums for better guidance. There are certain folks in here who know the info as well, but the forums here move so fast…. At least over there you will get more focused ZFS “experts”. Level1techs forum is also a good place for this as Wendell who runs it is all about pushing the envelope and is actively working to push ZFS and nvme forward, where sometimes the folks in the truenas forum are stuck in the past. 

[LIGISTX]

3 minutes ago, RONOTHAN## said:

I'm just gonna say, unless you really know what you're doing, you should probably hire a professional. 

 

Also, 10G networking is able to be saturated by HDDs, without too much effort as well. Just make sure to get a lot of RAM for the system. Half the reason LTT went all M.2 is because they really like overkill, and they're also using 40Gb networking to everyone, with dual 100Gb going to the server. You're no where near those speeds on a 10gig network. 

Agreed. For a 20 person architecture firm, they should easily be able to bring in a professional who will set this up correctly. OP, not that you can’t, but I know a fist bit about the topic and I would never put myself in a position to spec and deploy a company’s infrastructure like this. I would work with people who do this day in and day out, use my knowledge to really drive the correct requirements, and use them to make sure it all actually works. 
 

Lawrence systems (they have a great YouTube channel) does this type of stuff, and in every video he always says link in description to hire us. I only mention him because I know most of my networking and ZFS knowledge from his videos, but his company may be worth reaching out to. 

[BenzyVenue]

5 minutes ago, LIGISTX said:

Oh. I read m.2 as nvme. Will they be nvme or SATA m.2? If sata, you won’t have the extent of issues I mentioned above.

 

But either way, ECC should be used, and you will likely want 64GB of RAM at a minimum. But again, I’d look at truenas forums for better guidance. There are certain folks in here who know the info as well, but the forums here move so fast…. At least over there you will get more focused ZFS “experts”. Level1techs forum is also a good place for this as Wendell who runs it is all about pushing the envelope and is actively working to push ZFS and nvme forward, where sometimes the folks in the truenas forum are stuck in the past. 

Probably nvme m.2s. They had a windows server set up with HDDs before, but it was slow for their needs, and it seemed like the 10G networking was being bottlenecked by HDD speeds. I also know a fair amount of Linux, so I think I will go with the ZFS and TrueNas route. Thanks!

[Eigenvektor]

21 minutes ago, BenzyVenue said:

BTW, the reason I asked this in the first place was since I am helping a friend start up a business of about 20ish people, on a 10gig network. (We have all the networking switches and whatnot ready to go). This is an architectural company, so each Revit file is about 0.5 gb. So I am planning on making a server with M.2s which the employees work directly off of.

Ideally you'd run some benchmarks to determine the performance requirements of the software in question before buying anything. Working on some files remotely (maybe with local caching) is most likely a far cry removed from working on huge video files. Don't buy "bigger everything" based on guesswork. Make sure you know where the bottleneck is and how to fix it.

 

Also keep in mind that LTT is very often sponsored so they can "afford" to go way overboard when a smaller system would probably perform "good enough" for someone on a budget.

 

I agree with the others. I would leave this to someone who is experienced with building such systems. Doing this as a favor will only come back to bite you, because you'll now be stuck in a tech support role for something that was never officially agreed on, no contracts, no compensation, etc.

[LIGISTX]

4 minutes ago, BenzyVenue said:

Probably nvme m.2s. They had a windows server set up with HDDs before, but it was slow for their needs, and it seemed like the 10G networking was being bottlenecked by HDD speeds. I also know a fair amount of Linux, so I think I will go with the ZFS and TrueNas route. Thanks!

I would seriously consult with an appliance supplier before you purchase anything. This isn’t just a simple homelab config. 
 

Also, from my current understating, the way nvme interfaces with the truenas BSD kernel is a bit… not great. I don’t know the ins and outs of it well enough to attempt to even explain it, but this is why I say post on truenas forums at a minimum. 

[BenzyVenue]

Just now, LIGISTX said:

I would seriously consult with an appliance supplier before you purchase anything. This isn’t just a simple homelab config. 
 

Also, from my current understating, the way nvme interfaces with the truenas BSD kernel is a bit… not great. I don’t know the ins and outs of it well enough to attempt to even explain it, but this is why I say post on truenas forums at a minimum. 

Yea. I will def consult with other people abt this. I'll go post on the truenas forum rn.

[LIGISTX]

1 minute ago, BenzyVenue said:

Yea. I will def consult with other people abt this. I'll go post on the truenas forum rn.

The makes of truenas is netgate and they sell appliances… someone like Lawrence systems would work with you to properly spec and config an alliance for your needs. But what’s vital there is it’s all certified hardware and comes with support…