NAS Build Advice (Micro-ATX)

[TheCalamoose]

Budget (including currency): $500

Country: USA

Games, programs or workloads that it will be used for: NAS only

Other details:

 

Hello All,

 

I was hoping I could get some advice. I am new to building servers. 
 

I want to build a TrueNas Scale ZFS NAS (I have a separate Proxmox server for running apps). 
 

I have an Intel i5-10500 with an ASUS Prime H510M-E and will be using the CX3702 by Sliger, whenever it gets released. 
 

I would love to have mirrored VDEVs and a mirrored SLOG and an L2 ARC.
 

On the motherboard I have a PCIE 3.0 x16 and two PCIE 3.0 x1 slots and a single M.2 that shares with one of the four SATA ports.

 

Any recommendation on cards or anything as ideally I would love to use the x16 slot for dual SFP+ for high speed to my aggregation switch. 
 

Any advise?

 

[Electronics Wizardy]

25 minutes ago, TheCalamoose said:

I want to build a TrueNas Scale ZFS NAS (I have a separate Proxmox server for running apps). 

Gotta ask, why not add drives to the proxmox system, and make it the nas?

 

How much usable storage do you need?

 

26 minutes ago, TheCalamoose said:

 

On the motherboard I have a PCIE 3.0 x16 and two PCIE 3.0 x1 slots and a single M.2 that shares with one of the four SATA ports.

I'd probably use the x1 for ethernet, and the x16 for a hba to connect all the drives. Not optimal, but probably the best here. A board with more slots would be better.

 

27 minutes ago, TheCalamoose said:

I would love to have mirrored VDEVs and a mirrored SLOG and an L2 ARC.

What are you storing on the NAS, for most home nas use a slog and l2arc won't help much.

 

 

Does your budget include the drives? Case? That case isn't cheap, and I'd rather get a used supermicro case typically.

 

[LIGISTX]

49 minutes ago, TheCalamoose said:

I would love to have mirrored VDEVs and a mirrored SLOG and an L2 ARC.

Why? What is your use case? You almost certainly don't need L2 ARC, and still mostly certainly don't need a SLOG device. 

 

What is your networking speed, this is probably the most important question...

[TheCalamoose]

9 minutes ago, Electronics Wizardy said:

Gotta ask, why not add drives to the proxmox system, and make it the nas?

The Proxmox server is sort of my tinkering/lab server, so it occasionally goes offline and I need somewhere where backups and my wife can access the data basically always (aka the NAS). 
 

17 minutes ago, Electronics Wizardy said:

How much usable storage do you need?

I like buying 2 8TB HDD at a time so with the sliger I could have 10 drives mirrored will give me 40TB. But for now I am planning on having about 16TB usable. 
 

19 minutes ago, Electronics Wizardy said:

I'd probably use the x1 for ethernet, and the x16 for a hba to connect all the drives. Not optimal, but probably the best here. A board with more slots would be better.

Would you recommend bonding the NICs? If I put one in each of the x1 do you think it will do anything or more likely shares bandwidth?

 

20 minutes ago, Electronics Wizardy said:

What are you storing on the NAS, for most home nas use a slog and l2arc won't help much.

Totally agree, more want to do it for fun and learning than anything. If I was able to bond the SFP+ ports I would be able to make use of the speeds (in the rare occasion it would ever be done)

 

22 minutes ago, Electronics Wizardy said:

Does your budget include the drives? Case? That case isn't cheap, and I'd rather get a used supermicro case typically.

Budget does not include the case, that’s separate. The budget would be for any more drives or cards. I currently have two 8TB drives and a M2 1TB name and a 1 TB 2.5” SSD. 

[TheCalamoose]

1 minute ago, LIGISTX said:

Why? What is your use case? You almost certainly don't need L2 ARC, and still mostly certainly don't need a SLOG device. 

 

What is your networking speed, this is probably the most important question...

It’s overkill but I like to learn and optimize systems. My backend network could definitely support those speeds, though rarely ever needed. Again it’s more learning and then applying any micro concepts learned while overseeing IT projects for the UHNW individuals and SMBs I provide back office support service for professionally. 

[LIGISTX]

1 hour ago, TheCalamoose said:

optimize systems

I think we may have a different meaning of the word optimize. I am an engineer, and I run my life via optimization... I could easily throw some optane SLOG's and very fast L2ARC into my truenas box, but I know that my 10gig networking is not fast enough, nor is my workload one that would need this... I run 10x4TB drives in Z2, and I can write to the array at over 5 gb/s, and read at about the same, unless the data I need is in ARC which then obviously it maxes out the 10 gig link. My setup is optimized for my needs, throwing hardware at it doesn't make it more optimized, it makes its less optimized.

 

The only thing I have concindered doing is adding some optane for special metadata... that would make a very noticable differnece in how fast the NAS feels since it would be able to pull metadata much quicker, and it reducdes the number of reads to pull data off the drives in a big way.

 

All this said... before you go to L2ARC, you need to maximize actual L1ARC. How much RAM do you intend to put in this system? I am assuming 32 GB? L2ARC eats up actual ARC just to index what is in L2, so until you have 100's of GB's of RAM, which 1) means you likely have a need for more ARC, 2) have some RAM to spare for L2 indexing, sure, throw some L2 in it. But something else to remember, for almost all home workloads, L2 ARC will never actually be used anyways - once again, not optimizing...

 

I am all for learning, thats the entire point of homelab. But imo, a big part of that learning is learning what matters and what doesn't matter, and building systems to a spec that makes sense. Saying you want to add L2ARC and SLOG devices to a home NAS is a lot like going into a "new build thread" post linking to the most expensive components on a newegg page without actually knowing why you picked them. 

 

Obviously, trying to build out a capable sysetm as a business expense/self learning expendature to better your IT skills is worthwhile and valuable. So, if that is truely the case, fair enough.

 

Just my .02...

[Electronics Wizardy]

2 hours ago, TheCalamoose said:

I like buying 2 8TB HDD at a time so with the sliger I could have 10 drives mirrored will give me 40TB. But for now I am planning on having about 16TB usable. 

Makes sense, but I'd be tempted to go 16TB or bigger drives if you can.

 

2 hours ago, TheCalamoose said:

Would you recommend bonding the NICs? If I put one in each of the x1 do you think it will do anything or more likely shares bandwidth?

 

I don't see a point in bonding here. Performance of those HDDS is unlikely to be limited by 10gbe in most uses.

 

2 hours ago, TheCalamoose said:

Totally agree, more want to do it for fun and learning than anything. If I was able to bond the SFP+ ports I would be able to make use of the speeds (in the rare occasion it would ever be done)

 

If the board has m.2 slots might as well use them for l2arc and see the hit ratio(its probably gonna be low though).

 

I'd skip slog unless your doing sync writes.

 

2 hours ago, TheCalamoose said:

Budget does not include the case, that’s separate. The budget would be for any more drives or cards. I currently have two 8TB drives and a M2 1TB name and a 1 TB 2.5” SSD. 

Then that $500 should go pretty far here. PSU + HBA + 10GBE nic should all fit under 500 easily.

 

 

[TheCalamoose]

7 hours ago, LIGISTX said:

I think we may have a different meaning of the word optimize. I am an engineer, and I run my life via optimization... I could easily throw some optane SLOG's and very fast L2ARC into my truenas box, but I know that my 10gig networking is not fast enough, nor is my workload one that would need this... I run 10x4TB drives in Z2, and I can write to the array at over 5 gb/s, and read at about the same, unless the data I need is in ARC which then obviously it maxes out the 10 gig link. My setup is optimized for my needs, throwing hardware at it doesn't make it more optimized, it makes its less optimized.

 

The only thing I have concindered doing is adding some optane for special metadata... that would make a very noticable differnece in how fast the NAS feels since it would be able to pull metadata much quicker, and it reducdes the number of reads to pull data off the drives in a big way.

 

All this said... before you go to L2ARC, you need to maximize actual L1ARC. How much RAM do you intend to put in this system? I am assuming 32 GB? L2ARC eats up actual ARC just to index what is in L2, so until you have 100's of GB's of RAM, which 1) means you likely have a need for more ARC, 2) have some RAM to spare for L2 indexing, sure, throw some L2 in it. But something else to remember, for almost all home workloads, L2 ARC will never actually be used anyways - once again, not optimizing...

 

I am all for learning, thats the entire point of homelab. But imo, a big part of that learning is learning what matters and what doesn't matter, and building systems to a spec that makes sense. Saying you want to add L2ARC and SLOG devices to a home NAS is a lot like going into a "new build thread" post linking to the most expensive components on a newegg page without actually knowing why you picked them. 

 

Obviously, trying to build out a capable sysetm as a business expense/self learning expendature to better your IT skills is worthwhile and valuable. So, if that is truely the case, fair enough.

 

Just my .02...

A lot of very valuable information in here, so thank you for that. Your right especially on a 10th gen system that maxes out at 128GB of RAM and a micro-atx with only two slots, it’s definitely limiting. 
 

That’s why I posted on here because I am trying to learn, obviously I do not have everything purchased but if you think I should be just focused on RAM and drives instead of a SLOG and L2ARC, then that’s very helpful. 
 

Would you recommend using the x16 slot for an HBA or as dual SFP+ ports or what should I do for the networking?

[TheCalamoose]

7 hours ago, Electronics Wizardy said:

Then that $500 should go pretty far here. PSU + HBA + 10GBE nic should all fit under 500 easily.

I actually have a SFX Lian Li 750W sitting around for this build, so that’s not included in the budget. 
 

I think the biggest question is best use of the PCIE lanes and how to get the fastest networking while getting ability to use all ten drive bays. 

[TheCalamoose]

8 hours ago, Electronics Wizardy said:

I'd skip slog unless your doing sync writes.

With having a separate Proxmox server running Plex and 4K remuxes, wouldn’t NFS make the most sense therefore it does sync writes?

[LIGISTX]

2 hours ago, TheCalamoose said:

Would you recommend using the x16 slot for an HBA or as dual SFP+ ports or what should I do for the networking?

If you are running bare metal, no real reason for an HBA… assuming the mobo has enough SATA ports for your needs.
 

If you wanted to virtualize truenas, that’s when an HBA becomes valuable… I run truenas under proxmox and pass the HBA to truenas so it has bare metal access to the drives.

 

For networking, that really depends on what your network looks like and how you want to do this. I run a connectX 3 in my homelab, it’s also passed through directly to truenas. I then run an Intel 10gig card in my PC (I forget the specific card off hand, it’s not melonox since the connectX 3 I previously used on win 10 doesn’t work with win 11…) and run a point to put fiber subnet. The rest of my network is standard gigabit, but that’s fine. I get the speed between my PC and truenas, and since Truenas is virtual under procmox, all my homelab services have virtual 10gig+ between VM’s anyways…

[TheCalamoose]

52 minutes ago, LIGISTX said:

If you are running bare metal, no real reason for an HBA… assuming the mobo has enough SATA ports for your needs.

Current mobo has 4 sata ports and if I use the M2 then only three are usable. 

[Electronics Wizardy]

3 hours ago, TheCalamoose said:

With having a separate Proxmox server running Plex and 4K remuxes, wouldn’t NFS make the most sense therefore it does sync writes?

You can turn off sync with NFS. I'd do that for a media server.

 

4 hours ago, TheCalamoose said:

I actually have a SFX Lian Li 750W sitting around for this build, so that’s not included in the budget. 
 

I think the biggest question is best use of the PCIE lanes and how to get the fastest networking while getting ability to use all ten drive bays. 

the x1 slots are kinda limiting here. I'd probably get this sata card https://www.amazon.com/BEYIMEI-Splitter-Profile-Controller-Expansion/dp/B099ZJ8V7W/ref=sr_1_3?crid=3ONTRUS96XWG0&dib=eyJ2IjoiMSJ9._Zb7_v94R5GaQwKMVETjde_7uwYH0ADkj5mH1S_r1h1d1wkEWtGMFxXKo54cv4j0eVln2-GLENvQHQfj6qBDzXjPGqoh0D1QYNDJq3kVDzri5ErESLffgmPVFODONReLZ4fcYFIaKdb_1TR3UXm-pTapkCEI37O6SEJJw6I8YhVJneQ4DJl4SMLWEV5JvSnTbsJG799G0eP49AS7XULe7zLUHbxinbgwHhESaC7B3kM.VOea4n2k0kAzYRtIJvykbNavofseZfEc88il5zkEshc&dib_tag=se&keywords=pcie%2Bx1%2Bsata%2B8%2Bport&qid=1717777068&sprefix=pcie%2Bx1%2Bsata%2B8%2Bpor%2Caps%2C196&sr=8-3&th=1

 

and then a 10gbe nic that uses a x4 or x8 slot. I can't seem to find a x1 10gbe nic at first glance and while I don't like this sata card, you don't have muchbetter options here.

[TheCalamoose]

1 hour ago, Electronics Wizardy said:

and then a 10gbe nic that uses a x4 or x8 slot. I can't seem to find a x1 10gbe nic at first glance and while I don't like this sata card, you don't have muchbetter options here.

Would it be better to do two x1 sata cards?

[Electronics Wizardy]

9 minutes ago, TheCalamoose said:

Would it be better to do two x1 sata cards?

Potientally, but it won't matter in the real world. I think there are often gen3 x1 cards so about 1GB/s for all the drives. This is about the same as the network speeds, so it won't affect transfer speeds, but it may help with rebuild/resilver times. With 8 drives at 1gB/s your limited to about 125mB/s per drives which is a bit limiting for HDDs.

[TheCalamoose]

30 minutes ago, Electronics Wizardy said:

Potientally, but it won't matter in the real world. I think there are often gen3 x1 cards so about 1GB/s for all the drives. This is about the same as the network speeds, so it won't affect transfer speeds, but it may help with rebuild/resilver times. With 8 drives at 1gB/s your limited to about 125mB/s per drives which is a bit limiting for HDDs.

If I use the dual SFP+ 10 gig card bonded together on this and my Proxmox server I could have 10 gig between the two devices, would it then make sense to use two different x1 sata cards?

[Electronics Wizardy]

6 minutes ago, TheCalamoose said:

If I use the dual SFP+ 10 gig card bonded together on this and my Proxmox server I could have 10 gig between the two devices, would it then make sense to use two different x1 sata cards?

I doubt your gonna benefit from dual 10gbe with these drives as your probably going to be disk limited anyways, but might as well go dual sata controllers unless you want an additional ssd or something else in that slot.

[TheCalamoose]

7 minutes ago, Electronics Wizardy said:

I doubt your gonna benefit from dual 10gbe with these drives as your probably going to be disk limited anyways, but might as well go dual sata controllers unless you want an additional ssd or something else in that slot.

That’s why I was thinking of throwing in the M2 as the slog and two SSDs 2.5” for L2ARC (mirrored) just to try to squeeze a little more performance but also to mess around with it. Thoughts?

[Electronics Wizardy]

6 minutes ago, TheCalamoose said:

That’s why I was thinking of throwing in the M2 as the slog and two SSDs 2.5” for L2ARC (mirrored) just to try to squeeze a little more performance but also to mess around with it. Thoughts?

No point in mirroring the l2arc as its a read only cache. All data on the l2arc is already on the main data drives. 

 

The l2arc doesn't really help for things like copying media here. 

[TheCalamoose]

46 minutes ago, Electronics Wizardy said:

The l2arc doesn't really help for things like copying media here

Oh so it doesn’t help with Plex or whatever reading? The plan was to have my qbittorrent on the proxmox and it would download and seed and then copy it over to the NAS and then it would delete it from the downloads after x amount of storage in a FIFO pattern. So I was thinking it could help speed up the writes as well. 

[Electronics Wizardy]

6 minutes ago, TheCalamoose said:

Oh so it doesn’t help with Plex or whatever reading? The plan was to have my qbittorrent on the proxmox and it would download and seed and then copy it over to the NAS and then it would delete it from the downloads after x amount of storage in a FIFO pattern. So I was thinking it could help speed up the writes as well. 

Maybe a little bit, but in my use of the l2arc it likely will help very little. Feel free to try it for your self though.

 

l2arc is ready only, and won't help with writes at all. ZFS doesn't really have a write cache option.

[LIGISTX]

43 minutes ago, TheCalamoose said:

Oh so it doesn’t help with Plex or whatever reading? The plan was to have my qbittorrent on the proxmox and it would download and seed and then copy it over to the NAS and then it would delete it from the downloads after x amount of storage in a FIFO pattern. So I was thinking it could help speed up the writes as well. 

Unless you have some pretty wild internet speeds, you don’t need l2arc to cache Plex media to seed…. You don’t need need to cache it to play back content. I actually have ARC fully disabled for my /media directory in TrueNAS…. No point in wasting RAM trying to cache huge, sequential, rarely accessed files. And even when they are accessed, they never need to be accessed at over gigabit speeds... my array, purely from the array without data in ARC, can still read at over gigabit speeds… remember, gigabit is only 125MB/s. A single harddrive can saturate a gigabit link, none the less a ZFS array of multiple drives especially for large sequential reads. Sure, if you start hammering the array with lots of IOPS things will start to slow down and cause issues, but rarely does a home NAS see use cases such as this. 
 

Have your /media directory mounted as NFS to your qbitorrent VM/container. Let it write and read directly to the TrueNAS directory, no reason to have it FIFO storage and shuffle stuff around. That way you can have things forever seeding, just enforce some arbitrary upload bandwidth limit that makes sense for your internet connection. 
 

 

[TheCalamoose]

16 hours ago, Electronics Wizardy said:

l2arc is ready only, and won't help with writes at all. ZFS doesn't really have a write cache option.

Gotcha. Thank you!

[TheCalamoose]

15 hours ago, LIGISTX said:

Unless you have some pretty wild internet speeds, you don’t need l2arc to cache Plex media to seed…. You don’t need need to cache it to play back content. I actually have ARC fully disabled for my /media directory in TrueNAS…. No point in wasting RAM trying to cache huge, sequential, rarely accessed files. And even when they are accessed, they never need to be accessed at over gigabit speeds... my array, purely from the array without data in ARC, can still read at over gigabit speeds… remember, gigabit is only 125MB/s. A single harddrive can saturate a gigabit link, none the less a ZFS array of multiple drives especially for large sequential reads. Sure, if you start hammering the array with lots of IOPS things will start to slow down and cause issues, but rarely does a home NAS see use cases such as this. 
 

Have your /media directory mounted as NFS to your qbitorrent VM/container. Let it write and read directly to the TrueNAS directory, no reason to have it FIFO storage and shuffle stuff around. That way you can have things forever seeding, just enforce some arbitrary upload bandwidth limit that makes sense for your internet connection. 
 

 

Thank you for all of this and everything before, this will be very helpful with not only this build but future builds.