home server I need advice with my upcoming new HomeServer. Can you help ?

[Tyraenel]

Hi guys.

I would like to build new home sever and I need some advice. 

 

Currently I have 3 separate home servers ( Truenas, wazuh+docker+VM, Debian workstation for Llama AI).

All of them are basically repurposed workstations and miniPCs and I would like to build one server to include it all ( I have issue with space and available outlets).

I would like to install proxmox and VM (with PCIE passthrough) all other functionallity.

But I am on budget so it will be repurposed parts/scavenging from current workstations/ebay/aliexpress.

 

What I already bought:

Intel Xeon E5-2699 V3 18/36 cores/threads

ASUS Z10PA-U8 - LGA2011-3 ATX Mainboard

128 GB DDR4 ECC Memory

4U server desktop case computer - basically server chasis for internal 24 HDDs, normal ATX MB and PSU

 

What I scavenged/ repurposed:

Corsair HX1500i - overkill but I got it for free.

Nvidia Quadro 6 GB ( PCIE passthough to VM for AI ) don't remember exact model - I will replace it in future but for basic models it is OK. 

Nvidia Quadro 2 GB as main display adapter (shared except AI VM) exact model doesn't matter.

Various HDD (8-12TBs) - 16 pieces

some 512-1TB SDD drivers ( as system drive and caches )

 

And main issue

I have PCIE 3.0 1x 20 port SATA card basically for free.

Can I use it for 18+ HDDs/SSDs for Truneas VM with PCIE passthrough ?

Or should I buy 8x/16x version of the SATA card ? 

Or should I buy LSI HBA card + expander ? My issue is I didn't use LSI cards before and I don't know how they works ? Do I need PCIe Lanes for expander cards ? Can it be used with VM and PCIe passthrough ? and if yes which one should I buy ? Which expander? 

 

Also for truenas, if I use VDEVs with different capacity (mixed) for example 3x( 4x12TB) and 1x(4x8TB), how it affects pool. Would be better to create 2 pools and separate 12TB and 8TB drives ?

I have only 1GBit network and I want to use truenas for back up maaaybe some media server,( I will probably update it in next 2 years but currently 2.5GBit and 10GBit routers/switches are too expensive/big to use them)

Can someone explain it to me in detail ?

 

Thank you very much.

[Motifator]

You don't need a lot of bandwidth for HDDs, generally speaking, at least on tech data. They likely won't WANT to saturate the bandwidth like SSDs... that's granted. However, you're also asking to drive a lot of drives off a x1 slot controller which might not even have staggered spin up.

The bigger choir of the issue here is probably that controller card not having staggered spin-up, which means all the HDDs will spin their drivers up concurrently, hitting the PSU hard at cold boot. Sure, it's a 1500i... beast of a PSU, but still. With that many drives, you might as well just get one of those LSI controllers on the cheap, which would most likely have staggered spin up.

[LIGISTX]

28 minutes ago, Tyraenel said:

Or should I buy LSI HBA card + expander

Yes, do this. On ebay you can find them already flashed into the required IT mode. This is required for use with truenas since IT mode turns it into a simple HBA and will pass all the drives to truenas for ZFS to manage.

 

SAS expander needs a PCIe lane, but only for power, no data goes through its PCIe slot. I believe my expander can use molex power instead of pcie if you need to get creative...

 

I run proxmox and I pass the HBA through to truenas, this is who you want to do it. You must pass the entire PCIe device through to truenas so ZFS can see the drives directly. You will need to enable IOMMU: https://forum.proxmox.com/threads/where-to-add-intel_iommu-on-and-iommu-pt.131592/post-578296

 

32 minutes ago, Tyraenel said:

I have only 1GBit network and I want to use truenas for back up maaaybe some media server,( I will probably update it in next 2 years but currently 2.5GBit and 10GBit routers/switches are too expensive/big to use them)

You could set up a 10gig point to point network between truenas and your main PC. You would need 2 QFSP+ NIC's, and you can get fiber trasnceivers and run fiber between truenas and PC. I do this, and with the homelab in my signature (ZFS 10x4TB drives in Z2) I can read and write at about 5 gigabit (if the data I am trying to read is in ARC, then I obviously get full 10gig reads).

 

Hope this helps - I would say you have some more forum reading and learning to do before you dive to deep :). Definitely don't hesitate to ask more questions, but with the info I provided you should be able to start googling around and finding more info. Homelab and virutalization is the type of thing you really need to put the time into and ask questions when you are unsure - it isn't something any single person can just "tell you exactly how to do" :).

[Tyraenel]

30 minutes ago, Motifator said:

You don't need a lot of bandwidth for HDDs, generally speaking, at least on tech data. They likely won't WANT to saturate the bandwidth like SSDs... that's granted. However, you're also asking to drive a lot of drives off a x1 slot controller which might not even have staggered spin up.

The bigger choir of the issue here is probably that controller card not having staggered spin-up, which means all the HDDs will spin their drivers up concurrently, hitting the PSU hard at cold boot. Sure, it's a 1500i... beast of a PSU, but still. With that many drives, you might as well just get one of those LSI controllers on the cheap, which would most likely have staggered spin up.

Thank you, Do you think that the hit for the PSU can be too much ? 

[Tyraenel]

18 minutes ago, LIGISTX said:

Yes, do this. On ebay you can find them already flashed into the required IT mode. This is required for use with truenas since IT mode turns it into a simple HBA and will pass all the drives to truenas for ZFS to manage.

 

SAS expander needs a PCIe lane, but only for power, no data goes through its PCIe slot. I believe my expander can use molex power instead of pcie if you need to get creative...

 

I run proxmox and I pass the HBA through to truenas, this is who you want to do it. You must pass the entire PCIe device through to truenas so ZFS can see the drives directly. You will need to enable IOMMU: https://forum.proxmox.com/threads/where-to-add-intel_iommu-on-and-iommu-pt.131592/post-578296

 

You could set up a 10gig point to point network between truenas and your main PC. You would need 2 QFSP+ NIC's, and you can get fiber trasnceivers and run fiber between truenas and PC. I do this, and with the homelab in my signature (ZFS 10x4TB drives in Z2) I can read and write at about 5 gigabit (if the data I am trying to read is in ARC, then I obviously get full 10gig reads).

 

Hope this helps - I would say you have some more forum reading and learning to do before you dive to deep :). Definitely don't hesitate to ask more questions, but with the info I provided you should be able to start googling around and finding more info. Homelab and virutalization is the type of thing you really need to put the time into and ask questions when you are unsure - it isn't something any single person can just "tell you exactly how to do" :).

Thank you. 

 

Can you recommed which LSI card I should get and which expander ? ( I assume that you need to buy matching expander). 

I would need to get creative because the selected MB doesn't have much PCIe slots and I want to use 2 GPUs  and possible the QFSP NIC.

 

Good point for the point to point network

if I think about this solution, my idea is to have my main PC on my network and on point to point network  from PC to TrueNAS VM. And for TrueNAS VM I need the access to the local network ( so I can add SMB folder to synology NAS file structure for internet access ) and the point to point network. So it would mean I would need 1x  1GBit NIC passthrough for network ( there are 3 NICs on MB so no problem ) and 1 QFSP NIC for PCIe. Could it work ? Can I also attach SCSI connection/share via QFSP ? 

 

Again Thanks. 

[LIGISTX]

5 minutes ago, Tyraenel said:

So it would mean I would need 1x  1GBit NIC passthrough for network ( there are 3 NICs on MB so no problem )

You don’t need to pass NIC’s through like this. Proxmox will create virtual NIC’s for each VM, each gets its own IP on your subnet, it works great. No need to over complicate that part of it. I run almost 15 VM’s on a single 1 gig NIC on my mobo. Sure, if you have lots going on network transfer wise, that could be an issue, but for almost all home setups, this won’t be an issue especially if you set up point to point 10gig from main workstation to TrueNAS. Also, using the virtIO virtual NIC for the VM’s, proxmox actually just moves data from VM to VM internally and won’t even go out over the physical subnet. So you can easily get 10gig between VM’s even if you only have 1 gig physical connected to proxmox. 
 

11 minutes ago, Tyraenel said:

Can you recommed which LSI card I should get and which expander ?

I run dell H310’s. This one is already flashed to IT mode:
 

https://www.ebay.com/itm/155421555013?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=zq9OX4YXQYW&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

SAS expander I use:

 

https://www.ebay.com/itm/144481548648?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=OwHu05S9RzG&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

There are a few options, and I would do some research as there may be newer models that would make more sense to buy today. I bought all of this almost 8 years ago at this point, but it works perfect for me so I have not needed to think about this since I bought it all. 
 

SAS to SATA cable since you will need an extra to get 12 drives working. They also sell 2 packs so could always do that instead:

 

https://www.ebay.com/itm/153572967108?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=jFynvEpdSpS&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

 

To connect HBA to SAS expander:

 

https://www.ebay.com/itm/164322850606?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=CvlQ154ETKC&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

[Tyraenel]

17 minutes ago, LIGISTX said:

You don’t need to pass NIC’s through like this. Proxmox will create virtual NIC’s for each VM, each gets its own IP on your subnet, it works great. No need to over complicate that part of it. I run almost 15 VM’s on a single 1 gig NIC on my mobo. Sure, if you have lots going on network transfer wise, that could be an issue, but for almost all home setups, this won’t be an issue especially if you set up point to point 10gig from main workstation to TrueNAS. Also, using the virtIO virtual NIC for the VM’s, proxmox actually just moves data from VM to VM internally and won’t even go out over the physical subnet. So you can easily get 10gig between VM’s even if you only have 1 gig physical connected to proxmox. 
 

I run dell H310’s. This one is already flashed to IT mode:
 

https://www.ebay.com/itm/155421555013?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=zq9OX4YXQYW&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

SAS expander I use:

 

https://www.ebay.com/itm/144481548648?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=OwHu05S9RzG&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

There are a few options, and I would do some research as there may be newer models that would make more sense to buy today. I bought all of this almost 8 years ago at this point, but it works perfect for me so I have not needed to think about this since I bought it all. 
 

SAS to SATA cable since you will need an extra to get 12 drives working. They also sell 2 packs so could always do that instead:

 

https://www.ebay.com/itm/153572967108?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=jFynvEpdSpS&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

 

To connect HBA to SAS expander:

 

https://www.ebay.com/itm/164322850606?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=CvlQ154ETKC&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

Thanks. 

 

So just for the confirmation: 

1. I will add HBA card 

2. Add expander card

3.Connect HBA card with expander card with SFF 8087 to SFF8087 cable. 

4. Connect SFF8087 to 4 sata cables to extender and connect to drivers.

 

I assume that I can connect 2 extenders to the card to populate 2x20 HDD/SDDs.

Is that correct ? 

 

I pasted my cart from ebay ( I changed some things due to the postal fees or they didn't deliver to my country) 

Is it ok ? 

 

I am very sorry if I am bothering you. I just want to be 100% sure . 

 

Thanks Again.

[LIGISTX]

1 hour ago, Tyraenel said:

Thanks. 

 

So just for the confirmation: 

1. I will add HBA card 

2. Add expander card

3.Connect HBA card with expander card with SFF 8087 to SFF8087 cable. 

4. Connect SFF8087 to 4 sata cables to extender and connect to drivers.

 

I assume that I can connect 2 extenders to the card to populate 2x20 HDD/SDDs.

Is that correct ? 

 

I pasted my cart from ebay ( I changed some things due to the postal fees or they didn't deliver to my country) 

Is it ok ? 

 

I am very sorry if I am bothering you. I just want to be 100% sure . 

 

Thanks Again.

No problem at all. I would just confirm on truenas forums that the 9220-8i works fine, or maybe @Electronics Wizardy will know off hand (I don't, I assume it is fine though).

 

Second, you don't need 2 expanders. You can connect the HBA and the expander with a single cable, plug one of the SAS -> SATA cables into the HBA itself (which gives you 4 SATA drives), and then you have 5 open ports on the expander which would give you 20 more SATA devices if you fully populated the expander with SAS -> SATA cables.

 

Third, the HBA comes with 2 SAS -> SATA cables it looks like, so you would only need to buy 3 more, not 6, to get a total of 5 which is 20 total SATA drives.

 

Fourth, you would only need 1 SFF 8087 to SFF 8087 cable in this configuration.