looking to build m.2 only server

[TimedPing]

1 minute ago, TimedPing said:

Well, if your budget is around $1200 - $1500 including the drive's, is really thight actually, and I dont wanna recommend an used drive even you didn't care about the data loss, here's my specs sugestion for your needs:

1. Main hardware: you can choose between WS-prebuilts like HP Z840 or the better solution, DIY system, and heres the quick checkboxes you will need to follow:
   • The motherboard need to support x4x4x4x4 bifurfication, both HP Z840 and almost all Intel C612-based chipset motherboard support this, some chinese motherboard like the Huananzhi X99-TF/X99-TF-Q also supported it (for more info on Chinese Aliexpress motherboard PCIe bifurfication compability and durabilty you can visit Myconst hardware youtube channel or on other YouTube channel/sites).
   • Make sure the motherboard have good PCIe slot arrangement to properly get all the PCIe slot being used and slightly improve cooling, the Huananzhi X99-TF is a good example for that.
   • You need atleast 32GB of RAM, especially when you using TrueNAS/Proxmox and running it as hypervisor, however, I recommend 64GB ECC Fully Buffered RAM running in quad channel. sadly the Haswell-EP only support maximum of 2133-2400 MT/s RAM (hey atleast you got quad-channel as the tradeoff). and since mainstream server is began to replacing their old DDR4 to DDR5-based system, you can get the DDR4 ECC RAM pretty cheap in used condition.
   • USE GOOD POWER SUPPLY! You know why. This system config usually comfy running with good 550-750W PSU.
   • For the processor, you can choose between E5-2650/2660v3 (10C/20T) / E5-2680v3 (12C/24T), or if you wanna maxxed out with the cores, you can get E5-2699v3 (18C/36T) with more power hungry and premium price tradeoff. I personally use E5-2650v3 on my so called "home server".
   • For the CPU cooler, just make sure it can handle the CPU TDP you're using and the most important, MAKE SURE THE COOLER SUPPORT LGA-2011 SOCKET.
   • Remember that this platform doesn't have integrate GPU, make sure you have dedicated GPU for accessing the OS console, like GT 710/HD 5450 will do just fine. Some motherboard like Supermicro one have basic GPU provided by it's IPMI/iKVM chip (usually ASPeed AST2500/2600, etc).
2. Boot storage: you can use reputable brand 128GB SATA-based SSD in mirrored mode for redudancy purposes, personally use x2 128GB Teamgroup GX2 SSD.
3. L2ARC/SSD Cache/SSD VM storage: This part is the reason you will need x4x4x4x4 PCIe bifurfication support, while TrueNAS/Proxmox usually using RAM as storage cache (CMIW), you can add 2nd tier cache with faster NVMe drive. Remember, despite all of that, you need to buy PCIe x16 to 4x NVMe adapter to this to work, usually cost $25 - $40 on AliExpress/eBay. More explanation on SSD configuration as follows:
   • If you want use it as cache, I really recommend used 4x 64GB Intel optane M10 SSD, while yes, it's have slower throughput because only using 2 lanes of PCIe 3.0 bandwidth, but the price and random IO performance value cannot be matched especially when you running  it on RAID0
   • If you want use it as fast storage (for VM boot drive for example), you can choose several matches NVMe drive, like 4x 256GB Samsung 970 Pro running in RAID-Z1.
   • Remember most LGA 2011v3 motherboard have 2 PCIe x16 slot actually running in x16 mode, not like most consumer platform. So you if you have the additional budget, you can use both configuration mentioned above.
4. Bulk storage: You can use various config on this one, for example using 4x 4TB WD RED CMR drive to achieve 12TB usable storage with RAID-Z1 config. To save a few bucks, you can use refurbished enterprise drive or buy WD MyBook external drive which usually go on sale, on some of this drive, you need to cut off the 3.3v power on SATA Power lane, especially on salvaged WD MyBook HDD to avoid the drive not detected by the system. In case of your motherboard SATA is not enough, you can buy used Dell PERC H310 or other LSI SAS2008 based card flashed in IT mode to expand the SATA port (you get additional 8x SAS/SATA 6Gbit port on this card).
5. GPU(s): For Plex/Jellyfin video transcoding acceleration, you can use multiple option, but atleast make sure there's still available PCIe slot for it. If you're using low profile case, Video card like low profile GTX 1050Ti/GTX 1650 LP (Low Profile) is pretty nice. You just need to passtrough the GPU to the Plex/Jellyfin VM.
6. Other stuff: well, you can choose whatever case you want, just make sure all the component is fits and properly cooled, and since you want it quiet, you can find several budget PWM fan that suit the price.

Sorry for the long explanation, I hope my explanation is helping your journey to find the prefect system for your need. Also sorry for my bad english, I'm not native english speaker.

 

I forgot to mention the network card, you can add 2.5Gbit ethernet adapter or even better, an 10Gbit ethernet/SFP+ card. 1GBe ethernet definetly will peg the transfer performance.

[ZaziNabu]

8 minutes ago, TimedPing said:

I forgot to mention the network card, you can add 2.5Gbit ethernet adapter or even better, an 10Gbit ethernet/SFP+ card. 1GBe ethernet definetly will peg the transfer performance.

Wow, you just made a full on beginners guide on how to buy the first server! 
Your eng was amazing and you are amazing.
Thank you so much for the help, I'll look into all of it and see the costs and all.

 

[LIGISTX]

18 minutes ago, TimedPing said:

You need atleast 32GB of RAM, especially when you using TrueNAS/Proxmox and running it as hypervisor,

Without reading all of this… you really, really do not need that much RAM. 

 

I ran my homelab on 28GB of RAM for years, ESXi was my host, with truenas running as a VM next to multiple Ubuntu server’s, windows LTSC, home assistant, and a few other VM’s and a handful of docker containers. I gave truenas 16 GB of RAM, and it had 10x4TB in Z2…. All was perfectly fine and happy. 
 

More RAM is always good for ZFS, but for home use cases…. You really don’t need as much as people make it seem. 

[TimedPing]

1 minute ago, LIGISTX said:

Without reading all of this… you really, really do not need that much RAM. 

 

I ran my homelab on 28GB of RAM for years, ESXi was my host, with truenas running as a VM next to multiple Ubuntu server’s, windows LTSC, home assistant, and a few other VM’s and a handful of docker containers. I gave truenas 16 GB of RAM, and it had 10x4TB in Z2…. All was perfectly fine and happy. 
 

More RAM is always good for ZFS, but for home use cases…. You really don’t need as much as people make it seem. 

You 100% correct! well, since the price of DDR4 coming down, and those platform need quad-channel memory for better performance anyway, why not? Atleast 32GB of RAM is for futureproofing.   (Not mentioning if the OP want to use both the L2ARC and NVMe RAID option, the slow DDR4 that pegged to either 2133/2400MHz need to keep up, right?). I'm pretty sure in not so distance future the OP want to use this platform to other use case, so 32GB is pretty much great for that system.

[LIGISTX]

23 minutes ago, TimedPing said:

L2ARC/SSD Cache/

And OP doesn’t need L2ARC for this build. L2 will just increase the need of more RAM, but it won’t buy them anything in the way of performance for this system. 
 

You can always add L2 arc layer on if you deem it necessary, but it won’t be, not for this use case. 
 

Also of note, a 20+ thread xeon, on its own, would be plenty for all but the craziest Plex servers. I have a 2011 v3 based homelab as well, so I do support this idea… I give my Plex VM 6 threads of my 28 total, and it can transcode 5+ 1080p movies down to 720 at a time. So, no real need for a GPU here either.

 

Again, I am all for homelab. But going from 0 to 100 in one go is probably not the answer. The lga 2011-3 option is a good one, and eBay is the ideal place to source the parts. I got mobo, 64GB 2133 ECC, and Xeon all used, plus a brand new noctua 3U heatsink (which was 70 bucks) and a new nvme 980 boot drive (90 bucks) all for under 500 bucks, oh, and a used PCIe NVMe adapter as well. 
 

But, again, determine your requirements, and build the correct system for your needs. OP still needs to get their hands dirty and start determining what their needs are, which is part of the fun   

[TimedPing]

5 minutes ago, LIGISTX said:

And OP doesn’t need L2ARC for this build. L2 will just increase the need of more RAM, but it won’t buy them anything in the way of performance for this system. 
 

You can always add L2 arc layer on if you deem it necessary, but it won’t be, not for this use case. 
 

Also of note, a 20+ thread xeon, on its own, would be plenty for all but the craziest Plex servers. I have a 2011 v3 based homelab as well, so I do support this idea… I give my Plex VM 6 threads of my 28 total, and it can transcode 5+ 1080p movies down to 720 at a time. So, no real need for a GPU here either.

 

Again, I am all for homelab. But going from 0 to 100 in one go is probably not the answer. The lga 2011-3 option is a good one, and eBay is the ideal place to source the parts. I got mobo, 64GB 2133 ECC, and Xeon all used, plus a brand new noctua 3U heatsink (which was 70 bucks) and a new nvme 980 boot drive (90 bucks) all for under 500 bucks, oh, and a used PCIe NVMe adapter as well. 
 

But, again, determine your requirements, and build the correct system for your needs. OP still needs to get their hands dirty and start determining what their needs are, which is part of the fun   

Indeed, I agree to this one! I'm adding those options hoping that OP know what kind of option available for him to start, atleast since the OP mentioned that he/her is new to the NAS space (I'm also a newbie on this space, especaily on software side of things  ). Anyway, we can definetly agree that Haswell and Broadwell Xeons still have the power (atleast in the homelab space) and being effecient enough to run in 24/7. LONG LIVE HASWELL XEONS!

[ZaziNabu]

Good news! 

 

I'm going to get an old ibm server to abuse while I'm at work. 

 

I still need to find the right mb and parts, but I'll have a chance to start software wise till I'll get all the parts I need. 

 

For what I understand, the best option is a used xenon mb (that supports 4x4x4) and cpu, 32gb and up ecc ram, cooling, adapters, I'll use an old gpu I have (gtx1080) and new storage devices (as they degrade). 

 

For future usage I can add optane as l2 cache. 

 

Did I get the basics right? 

 

[LIGISTX]

1 hour ago, ZaziNabu said:

Good news! 

 

I'm going to get an old ibm server to abuse while I'm at work. 

 

I still need to find the right mb and parts, but I'll have a chance to start software wise till I'll get all the parts I need. 

 

For what I understand, the best option is a used xenon mb (that supports 4x4x4) and cpu, 32gb and up ecc ram, cooling, adapters, I'll use an old gpu I have (gtx1080) and new storage devices (as they degrade). 

 

For future usage I can add optane as l2 cache. 

 

Did I get the basics right? 

 

Those basics are correct. But I would hold off on the GPU until you know you need it. No point in burning electricity in parts that you don’t need. 
 

You likely won’t need L2arc…. At any point in time, if you end up needing more ZFS cache, get more RAM.

 

More importantly, start thinking through how you want to use this machine. Is this going to be a bare metal NAS, or will you run something like proxmox or xcp-ng as a hypervisor and rub your storage appliance virtually under that (this is what I and many others do). This is an important choice to make early on as migrating after the fact, while possible (I have done it… from bare metal to ESXi and ESXi to proxmox), it’s much easier to not have to migrate after stuff is set up. This will also drive some hardware choices, such as a SAS HBA you can then pass through to your storage VM. If you run bare metal, you can just use the SATA ports on the mobo, if you go virtual, you want to pass through a full PCIe HBA. 

[ZaziNabu]

6 minutes ago, LIGISTX said:

Those basics are correct. But I would hold off on the GPU until you know you need it. No point in burning electricity in parts that you don’t need. 
 

You likely won’t need L2arc…. At any point in time, if you end up needing more ZFS cache, get more RAM.

 

More importantly, start thinking through how you want to use this machine. Is this going to be a bare metal NAS, or will you run something like proxmox or xcp-ng as a hypervisor and rub your storage appliance virtually under that (this is what I and many others do). This is an important choice to make early on as migrating after the fact, while possible (I have done it… from bare metal to ESXi and ESXi to proxmox), it’s much easier to not have to migrate after stuff is set up. This will also drive some hardware choices, such as a SAS HBA you can then pass through to your storage VM. If you run bare metal, you can just use the SATA ports on the mobo, if you go virtual, you want to pass through a full PCIe HBA. 

Did not understand most of it lol, but I'll google the meaning and learn.

All I wanna do now on the NAS is 3 things:
- normal storage

- plex/media server

- steam cache server

 

I think I'll need vm's for all of it to work on one server? 

[LIGISTX]

Just now, ZaziNabu said:

Did not understand most of it lol, but I'll google the meaning and learn.

All I wanna do now on the NAS is 3 things:
- normal storage

- plex/media server

- steam cache server

 

I think I'll need vm's for all of it to work on one server? 

Yes. I would plan on VM’s. 
 

With knowing you plan on VM’s, I’d look into using proxmox as your hypervisor (this is the OS you instal on the boot drive). 

[ZaziNabu]

2 minutes ago, LIGISTX said:

Yes. I would plan on VM’s. 
 

With knowing you plan on VM’s, I’d look into using proxmox as your hypervisor (this is the OS you instal on the boot drive). 

I'll try it out on Sunday, I'll get the testing server at work by then  

 

[TimedPing]

6 hours ago, ZaziNabu said:

I'm going to get an old ibm server to abuse while I'm at work. 

 

You just want to use the case/chasis right? 

[Blue4130]

8 hours ago, TimedPing said:

Well, if your budget is around $1200 - $1500 including the drive's, is really thight actually, and I dont wanna recommend an used drive even you didn't care about the data loss, here's my specs sugestion for your needs:

1. Main hardware: you can choose between WS-prebuilts like HP Z840 or the better solution, DIY system, and heres the quick checkboxes you will need to follow:
   • The motherboard need to support x4x4x4x4 bifurfication, both HP Z840 and almost all Intel C612-based chipset motherboard support this, some chinese motherboard like the Huananzhi X99-TF/X99-TF-Q also supported it (for more info on Chinese Aliexpress motherboard PCIe bifurfication compability and durabilty you can visit Myconst hardware youtube channel or on other YouTube channel/sites).
   • Make sure the motherboard have good PCIe slot arrangement to properly get all the PCIe slot being used and slightly improve cooling, the Huananzhi X99-TF is a good example for that.
   • You need atleast 32GB of RAM, especially when you using TrueNAS/Proxmox and running it as hypervisor, however, I recommend 64GB ECC Fully Buffered RAM running in quad channel. sadly the Haswell-EP only support maximum of 2133-2400 MT/s RAM (hey atleast you got quad-channel as the tradeoff). and since mainstream server is began to replacing their old DDR4 to DDR5-based system, you can get the DDR4 ECC RAM pretty cheap in used condition.
   • USE GOOD POWER SUPPLY! You know why. This system config usually comfy running with good 550-750W PSU.
   • For the processor, you can choose between E5-2650/2660v3 (10C/20T) / E5-2680v3 (12C/24T), or if you wanna maxxed out with the cores, you can get E5-2699v3 (18C/36T) with more power hungry and premium price tradeoff. I personally use E5-2650v3 on my so called "home server".
   • For the CPU cooler, just make sure it can handle the CPU TDP you're using and the most important, MAKE SURE THE COOLER SUPPORT LGA-2011 SOCKET.
   • Remember that this platform doesn't have integrate GPU, make sure you have dedicated GPU for accessing the OS console, like GT 710/HD 5450 will do just fine. Some motherboard like Supermicro one have basic GPU provided by it's IPMI/iKVM chip (usually ASPeed AST2500/2600, etc).
2. Boot storage: you can use reputable brand 128GB SATA-based SSD in mirrored mode for redudancy purposes, personally use x2 128GB Teamgroup GX2 SSD.
3. L2ARC/SSD Cache/SSD VM storage: This part is the reason you will need x4x4x4x4 PCIe bifurfication support, while TrueNAS/Proxmox usually using RAM as storage cache (CMIW), you can add 2nd tier cache with faster NVMe drive. Remember, despite all of that, you need to buy PCIe x16 to 4x NVMe adapter to this to work, usually cost $25 - $40 on AliExpress/eBay. More explanation on SSD configuration as follows:
   • If you want use it as cache, I really recommend used 4x 64GB Intel optane M10 SSD, while yes, it's have slower throughput because only using 2 lanes of PCIe 3.0 bandwidth, but the price and random IO performance value cannot be matched especially when you running  it on RAID0
   • If you want use it as fast storage (for VM boot drive for example), you can choose several matches NVMe drive, like 4x 256GB Samsung 970 Pro running in RAID-Z1.
   • Remember most LGA 2011v3 motherboard have 2 PCIe x16 slot actually running in x16 mode, not like most consumer platform. So you if you have the additional budget, you can use both configuration mentioned above.
4. Bulk storage: You can use various config on this one, for example using 4x 4TB WD RED CMR drive to achieve 12TB usable storage with RAID-Z1 config. To save a few bucks, you can use refurbished enterprise drive or buy WD MyBook external drive which usually go on sale, on some of this drive, you need to cut off the 3.3v power on SATA Power lane, especially on salvaged WD MyBook HDD to avoid the drive not detected by the system. In case of your motherboard SATA is not enough, you can buy used Dell PERC H310 or other LSI SAS2008 based card flashed in IT mode to expand the SATA port (you get additional 8x SAS/SATA 6Gbit port on this card).
5. GPU(s): For Plex/Jellyfin video transcoding acceleration, you can use multiple option, but atleast make sure there's still available PCIe slot for it. If you're using low profile case, Video card like low profile GTX 1050Ti/GTX 1650 LP (Low Profile) is pretty nice. You just need to passtrough the GPU to the Plex/Jellyfin VM.
6. Other stuff: well, you can choose whatever case you want, just make sure all the component is fits and properly cooled, and since you want it quiet, you can find several budget PWM fan that suit the price.

Sorry for the long explanation, I hope my explanation is helping your journey to find the prefect system for your need. Also sorry for my bad english, I'm not native english speaker.

 

Based on what the OP plans on using this for, your suggestion seems like WAY overkill.

[TimedPing]

4 minutes ago, Blue4130 said:

Based on what the OP plans on using this for, your suggestion seems like WAY overkill.

well, my bad. I recommend that kind of platform because mainly it's supporting x4x4x4x4 PCIe bifurfication, having faster DDR4 running in quad channel, and certainly help if the OP upgrade it later down the road to all flash storage, tons of PCIe lanes, and since it's a haswell, it's running pretty effecient and most modern feature is supported (except AVX 512 and PCIe 4.0, but who cares this feature for OP purpose) all of that for ridicolously dirt cheap price (less than $30 for 8/10/12 cores and 40 PCIe lanes? heck yeah. used priced of course). I just giving OP list for lot of flexibility to play around with the system configuration down the road and all the option to upgrade it in the future.

[ZaziNabu]

Well, if it's cheaper then the budget (or fits the budget), I'll be able to upgrade it in the future and use it for the long run for more then what I needed it sounds good to me lol.

 

[TimedPing]

57 minutes ago, ZaziNabu said:

Well, if it's cheaper then the budget (or fits the budget), I'll be able to upgrade it in the future and use it for the long run for more then what I needed it sounds good to me lol.

 

From my experience running these system, Haswell-EP platform still have long life to go, I just stopped using older Nehalem based Xeon (e.g X5650) on 2020 for actual deployment as SMB NAS because the price of Haswell-EP have coming down and it's not longer efficient to run it 24/7, and also lacking modern CPU extension lake AVX, those CPU come out early as 2010. If intel keeping to make their server lineup like cascade lake who just launched, Haswell-EP xeons still have atleast 4-5 years to go (as long people keep making/selling motherboard for it).